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SYSTEM, APPARATUS AND METHOD FOR MAXIMIZING EFFECTIVENESS 
AND EFFICIENCY OF LEARNING. RETAINING AND RETRIEVING 
KNOWLEDGE AND SKILLS 

BACKGROUND OF THE INVENTION 

1. Field of the Invention 

The present invention relates to a systerti. apparatus and method for 
learning, and more specifically, relates to a system, apparatus and method for 
interactively dnd adaptively maximizing the effectiveness and efndency of 
learning, retaining and retrieving knowledge and skills including flexibly ' 
scheduling learning, retaining and retrieving knowledge and skills based on 
various factors and input information. 

2. DESCRIPTI ON OF BAr.KGRQUND ARJ 

Previous systems and methods for leanning have focused on 
presenting an item or items to be learned In a paired-associate format, such as 
a cue and response system. These priorart systems and methods have relied 
heavily on the motivation and metacognftive skills of the student and therefore, 
have varying degrees of effectiveness and efnciency. More importantly, such ' 
prior art methods and systems have very limited success in temis of a student 
actually acquiring knowledge or skills rapidly, ensuring that the student 
maintains the knowledge and skills to a high degree of retention for an 
extended period of time, and enabling the student to retrieve knowledge and 
skills automatically at some future date. 

In a well-knovwi prior art method, a paired-associate learning method is 
embodied in a group of flashcards. which may be presented manually or 
electronically via a computer, for example. In a typical example of such a 
method, a student starts by separating flashcards into two groups: known and 
unknown.. The student studies each unknown flashcard by first viewing the 
question on one side of the flashcard and then formulating a response to the 
question. The student then turns the card over and views the ansvi/er 
provided. The student judges the' adequacy of his response by comparing his 
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answer to the correct answer. If the student believes he has learned or 
"knows" the paired-associate, that flashcard is placed in the group of known 
items. When the student has studied all of the flashcards in the first unknown' 
group, and all of the flashcards have been transferred to the group of known 
5 flashcards; the student may review the group of known items in the same 
manner as described above. In ah alternative method, the cards can be 
shuffled for learning. Thus, in this method, the learning and review is 
performed by a student simply looking at flashcards to detemiine correct 
responses and reviewing the flashcards as desired, with no fixed schedule or 
10 sequence. 

In another method invented by B.F. Skinner, a method of learning and 
reviewing is provided. More specifically. Skinner discloses a machine which 
presents a number of paired-associate questions and answers. The learning 
machine has an area for providing questions, and an area where the user 
writes in an answer to these questions. At the time the question is presented, 
the correct answer is not visible. A student reads a question and then writes in 
an answer in the area provided. The user turns a handle which causes a clear 
plastic shield to cover his answer while revealing the correct answer. The user 
judges the adequacy of his response. If the user judges that his answer is 
adequate, he slides a lever which punches a hole in the question and answer 
sheet and turns a handle revealing the next question. If the answer is judged 
to be inadequate, the user simply tunns a handle revealing the next question. 
After all of the questions have been answered a first time, the user can make a 
second pass through the questions and answers. The machine operates such 
that only the questions which were answered incorrectly in the first pass are 
viewable during the second pass so as to provide a review of questions which 
were answered incorrectly. Thus, this conventional method provides a crude 
method of enabling review of missed questions. 

A slightly more advanced method was invented by Sebastian Leitner 
and described in "So Lernt Man Lernen." The method involves studying 
flashcards as in the method described above, but in addition, involves using a 
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specially constructed box to calculate review schedules. More specifically, the 
box has five compartments increasing in depth from the first compartment to 
the fifth compartment. According to Leitner's method, a student takes enough 
"unknown- flashcards to fill the first compartment and places them in the first 
compartment. The student begins by taking the first card out of the box and 
reading the question. The student then constructs an answer and compares it 
to the cdn-ect answer on the back of the card. If the student is correct the 
student places the flashcard In the second compartment. If the student cannot 
construct an answer, or If the student's answer is Incorrect, the student places 
the flashcard at the back of the group of cards in the first compartment. This 
process continues until all of the cards have been moved to the second 
compartment and the student stops the learning session. The next learning 
session begins by placing new "unknown" cards into the first compartment. 
The process of studying and sorting is performed as described above until 
once again, no cards remain in the first compartment. At some future date the 
second compartment will be full of cards placed there during previous learning 
sessions. At that time, the student begins to study the cards in the second 
compartment, except this time, known cards are placed into the third 
compartment and unknown cards are placed backed into the first compartment 
New cards are continually introduced into the first compartment and are moved 
through the compartments as they are learned and reviewed. Cards that are 
easily remembered or known are moved from the more shallow compartments 
to the deeper compartments and therefore are reviewed less and less 
frequently. Cards that are more difficult to learn are put back Into the more 
shallow compartments for more frequent review. This method provides a cmde 
form of scheduled review of learned items based on item difficulty. 

A computer-based version of Leitner's method is provided In the 
German language computer software program entitled Lemkartei PC 7.0 and 
in the Spanish language computer software entitled ALICE (Automatic 
Learning In a Computerized Environment) 1.0. With ALICE 1.0. question and 
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answer units are presented to a user and the number of cards and interval of 
time between study sessions are distributed to adapt to a user's work habits. 

Other conventional methods have recognized the importance of 
developing a system to present items for review. For example, a computer 
program developed by Piotr yVozniak in Poland and referred to as 
"SuperMemo" uses a mathematical model of the decline of memory traces to 
determine spacing of repetitions to maintain long term retention of paired- 
associates. 

In another prior art method described in U.S. Patent Nos. 5.545,044. 
5.437.553 and 5.577.919 issued to Collins et al.. paired-associates are 
presented to a user for learning. However, unlike the conventional methods 
described above, in this invention, the user is first queried as to whether a 
particular item is perceived to be known or unknown, not whether the user 
actually knows the item, or knows the con-ect answer to a question. That is, a 
user is asked to determine whether they think they know the correct response 
to the cue, not what the correct response actually is. Then, a sequence of 
perceived known items and perceived unknown items is generated and 
presented to the user in the form of cue and response for learning. Similar to 
the first conventional method described above, the question of the perceived 
20 known or unknown items Is presented to a user, the user constructs a 

response to the presented cue and then compares the constructed response to 
the correct response. 

TTie prior art methods described above have generally proven to be 
only marginally effective for learning, retaining and retrieving knowledge and 
skills. The prior art methods often require a user to schedule and manage the 
learn, review and test processes which consequently consumes a portion of 
the cognitive woriOoad of the user thereby reducing efficiency of learning, 
retaining and retrieving knowledge and skills. The cognitive workload is the 
amount of mental work that an organism, such as a human, can produce in a 
specified time period. By diverting some ofthe cognitive workload away from 
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learning, the organism is distracted from learning and cannot devote ail of the 
available cognitive resources to learning. 

Furthermore, because the user is making subjective judgements of 
perceived knowledge, they provide feedback to the method that is distorted by 
certain poghitive illusions inherent in self-paced study. These subjective 
inputs result in less effective learning than would otherwise be possible. 
Furthermore, even though some of the prior art methods monitor progress of 
learning or reviewing or testing, future learning or reviewing or testing are not 
modified based on a student's actual performance results. 

In addition, most prior art methods seek to train or teach knowledge or 
skills only to a level of recall in which a person or organism must expend 
significant cognitive effort to attempt to remember an item previously learned. 
Conventional methods have not been successful in training or teaching 
knowledge or skills to a level of automaticity in which performance is 
characterized by an extremely rapid response without conscious effort or 
attention. 

Also, there are many different theories, scientific principles, and 
concepts relating to learning, memory and perfomiance, which seek to explain 
how humans and other organisms are able to encode, store and retrieve 
knowledge and skills. Although these theories, principles and concepts have 
been studied, they have not been quantitatively measure and applied in a 
synergistic and effective manner to improve learning, reviewing and retrieving 
knowledge and skills. Furthennore, the prior art methods do not train a student 
to become a better learner by monitoring and Improving their metacognltive 
skills, but merely provide a marginal improvement in the ability to encode and 
recall learned items. 

SUMMARY OF THE INVENITinNl 

To overcome the problems described above and to provide other 
significant and previously unattainable advantages, preferred embodiments of 
the present invention provide a system including various apparatuses and 
methods for maximizing the effectiveness and efficiency of learning, reviewing 
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and retrieving knowledge and skills in an interactive and adaptive matter that 
minimizes the cognitive workload or mental effort of the user, while minimizing 
or eliminating forgetting of the knowledge or skills learned. 

In addition, prefen-ed embodiments of the present invention provide a 
system including apparatuses and methods for allowing a user to encode, 
store and retrieve knowledge and skills flexibly, and efficiently in response to 
overt and covert inputs from the user or other sources of Input Information, 
such as a desired level of memory strength, the date when the knowledge or 
skills are needed, the schedule of availability to use the system and many 
other factors. 

More specifically, preferred embodiments of the present invention 
provide a system including various apparatuses and methods for maximizing 
the ease of use of the system and maximizing the results of learning, retaining 
and retrieving of knowledge and skills by allowing a user, administrator or 
other input information source to interactively and flexibly input information to 
be learned, identify confusable items to be learned, select desired levels of 
initial learning and final retention of knowledge or skills, and input preferences 
regarding scheduling of learning, reviewing and testing and other input 
information relating to the learning, reviewing and testing of knowledge or 
skills. Based on these and other input infonnation, the system schedules 
operation of the learn, review and test operations in the most efficient way to 
guarantee that the user achieves the desired degree of learning within the 
desired time period. 

Furthermore, prefen-ed embodiments of the present invention provide a 
system Including apparatuses and methods which Include a Learn Module for 
presenting new knowledge or skills to a user, a Review Module for presenting 
previously learned knowledge or skills to a user in order to maintain a desired 
level of retention of the knowledge or skills learned previously^ and a Test 
Module for testing of previously learned knowledge or skills. Each of the three 
modules are preferably adapted to interact with the other two modules and the 
future operation of each of the Learn, 
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Review and Test modules and scheduling thereof can be based on previous 
performance in the three modules to maximize effectiveness and efficiency of 
operation. 

The advantages achieved by basing the interaction and scheduling of 
the Learn, Review, and Test Modules on previous performance in the three 
modules include achieving much more effective and efficient combined and 
overall operation of each of the three main modules so that a user encodes, 
stores and retrieves knowledge and skills much more effectively and efficiently, 
while also becoming a better learner. 

Also, preferred embodiments of the present invention provide a system 
including various methods and apparatuses which provide an extremely 
effective method of encoding, storing and retrieving knowledge or skills which 
are quantitatively based and interactively modified according to a plurality of 
scientific disciplines such as neuroscience (the scientific study of the nervous 
system and the cellular and molecular mechanisms associated with . learning 
and memory), cognitive psychology (an approach to psychology that 
emphasizes internal mental processes), and behavioral psychology (an 
approach to psychology that emphasizes the actions or reactions produced in 
response to extennal or internal stimuli), as well as scientific principles 
including: active recall (the process whereby a student constructs a response 
to a presented cue as opposed to passive recall in which a student simply 
observes a cue and response paired presented), the alternative forced-choice 
method (a test of memory strength sensitive to the level of recognition in which 
a c6e is presented followed by the con-ect response randomly arranged among 
several alternative choices called distracters, and in which the student must 
discriminate the correct response from the distracters), 
arousal (the student's experience of feeling more or less energetic which 
feeling is accompanied by physiological changes in perspiration, pupil 
diameter, respiration and other physiological reactions, and which influences 
information processing, in particular, the encoding and retrieval of information), 
attention (the ability or power to concentrate mentally by focusing on certain 
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aspects of current experience to the exclusion of all others), automaticity 
(performance characterized by rapid response without conscious attention or 
effort), the auditory rehearsal loop (the process of rehearsal, usually via 
subvocal speech, to maintain verbal Information in memory, in which the loop 
5 is capable of holding approximately 1 .5 to 2.0 seconds worth of information), 
classical conditioning (the procedure in which an organism comes to display a 
conditioned response to a neutral conditioned stimulus that has been paired 
with a biologically significant unconditioned stimulus that evoked an 
unconditioned response), cognitive workload (the amount of mental work that 

10 a student produces or can produce in a specified time period), confidence (a 
subjective judgement made regarding the degree of certainty of the 
correctness of a constructed response or of a subjective evaluation), 
consolidation (the initial period of time in memory formation when information 
in a relatively transient state is transformed to a more permanent, retrievable 

15 state), consolidation period (the interval during which the transformation to the 
more permanent retrievable state occurs), contiguity (two items occurring or 
being presented close together in time), contingency (two items being 
presented or occurring in a manner such that the occurrence of one item 
increases the probability that another item will occur, which is required to form 

20 a conditioned association), discrimination (the act of distinguishing between 
two or more items by noting the differences between the two or more items), 
ease of learning (a metacognitive judgement made in advance of knowledge 
acquisition in the form of a prediction about what will be easy or diffrcult to 
learn), encoding specificity (the theory that memory performance is better 

25 when tested in the presence of the same cues that were present when the 
memory wais formed), encoding variability (the theory that memory 
performance is better when multiple cues are available to generate a desired 
response), feeling of knowing (a metacognitive judgement made during or after 
knowledge or skill acquisition as to whether a given, currently non-recallable 

30 item is known or will be remembered on a subsequent retention test), 

generalization (when a response is evoked by a cue other than the one it was 



10 



15 



20 



25 



30 



WO 01/50439 

PCTAJSOO/35381 



9 

conditioned to), habituation (a decrease in response as a result of repeated 
exposed to 

a stimulus), instaimental conditioning (a situation in which a particular stimulus 
occurs and if an organism generates a response, then a particular reinforcer 
wiW occur), interference (a negative relationship between the learning of two 
sets of material), judgement of learning (a metacognitive judgement during or 
soon after knowledge acquisition which is a prediction about future test 
performance on currently recallable items), the labor-in-vain effect (In self- 
paced study, students make metacognitive judgements that determine the 
allocation of effort and often study beyond the point where any benefit is 
derived), latency of recall (a measure of time required to construct 
a response to a presented cue), learned helplessness (when a negative 
reinforcement is provided independent of a student's performance, the student 
behaves as though they have no control over their situation), long term 
potentiation (when appropriate stimulation is provided to some areas of the. 
brain, there is a long term increase in the magnitude of the response of the 
cells to further stimulation), memory activation (the availability of an item in 
memory such that items which have been recalled recently have relatively 
higher activation than those that have not), memory strength (a property 
of memory which increases with- repeated practice and is the degree to which a 
cue can activate a memory record), metacognition (the process of monitoring 
and controlling mental processes, particularly those associated with the acts of 
learning and retrieving), overleaming (learning that continues past the point 
where the student is first able to construct the correct response to a presented 
cue), paired-associate learning (a memory procedure in which the student 
learns to give a response when presented with a cue), performance (the 
observable qualities of learning; sometimes measured by the ability to 
discriminate a signal from noise), probability of recall (a measure of the 
likelihood that a student will be able to cohstnjct the correct response to a 
presented cue), rapid serial visual presentation (the presentation of a passage 
of text, one word or phrase at a time, serially, each in the same position on a 
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display, so as to increase reading speeds and eliminate saccades required in 
normal reading), rehearsal (the process of repeating information to oneself in 
order to remember it), reinforcement (following a behavior with an especially 
powerful event such as a reward or punishment), the retrieval practice effect 
(the act of retrieving an item from memory facilitates subsequent retrieval 
access of that item and the act of retrieval does not simply strengthen an item's 
representation In memory, it also enhances the retrieval process), savings in 
releaming (a measure of memory strength calculated by measuring the amount 
of time necessary to relearn an item to the same criteria as that attained in the 
initial learning session), sensitization (increase In response as a result of 
repeated exposure to a stimulus), the serial position effect (the observation 
that items at the beginning and end of a list that are learned in serial order are 
more easily remembered than Items In the middle of the list), signal detection 
theory (a method used to measure the criterion an observer uses in making 
decisions about signal existence and to measure the observer's sensitivity that 
is independent of his decision criteria), the spacing effect (the finding that for a 
given amount of study time, spaced presentations yield substantially better 
learned than massed presentations), the time of day effect (differences in 
perfomiance on learning tasks and other factors relating to circadian rhythms 
20 depending on the time of day), transfer appropriate processing (the concept 
that memory perfortnahce Is better when a student processes an item in the 
same way in which the Item was processed during learning or study), vigilance 
(the process of paying close and continuous attention). Von Restorff effect (the 
observation that an item from one category that is learned as a part of a serial 
25 list of items all from a different category will be more easily recalled than items 
from around it in the |ist) and many other important factors which are applied in 
novel and unique ways, both individually and in combination with other factors. 
For the first time, the above-listed factors or.phenomena are measured In a 
quantifiable manner and the measurements of the effects of these factors are 
30 used to interactively and adaptively modify the 
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processes of learning, reviewing and testing knowledge and skills to achieve 
results never before obtainable. 

While the above-listed factors have been studied in the past, and the 
effects thereof sometimes even measured, the measurements have not been 
quantified and then used in a feedback system to continuously and 
interactively modify future encoding, storage and retrieval of knowledge and 
skills to achieve maximum effectiveness arid efficiency. 

The system, apparatuses and methods of preferred embodiments of 
the present invention may be used td perfomi learning, reviewing and testing 
of any. type of knowledge and skills in any fonnat. The Information including 
knowledge or skills to be learned, reviewed and tested, referred to as 
"content," can be obtained from any source including but not limited to a text 
source, an image source, an audible sound source, a computer, the Internet, a 
mechanical device, an electrical device, an optical device, the actual physical 
world, etc. Also, the content may already be included in the system or may be 
input by a user, an administrator or other source of information. While the 
knowledge or skills to be learned, reviewed and tested may be presented in 
the form of a cue and response or question and answer in preferred 
embodiments of the present invention, other methods and formats for 
presenting items to be learned, reviewed and tested may be used. 

More specifically, the content is preferably arranged in paired- 
associate (cue and response) format for ease of learning. The paired- 
associates may be presented visually, auditorily, kinesthetically or in any other 
manner in which knowledge or skills can be conveyed. The content may be 
also an-anged in a serial or non-serial procedural order for skill-based learning. 
Any other arrangements where there is any form of a cue with an explicit or 
implicit paired response or responses are appropriate 

for use in the systems, methods and apparatuses of preferred embodiments of 
the present invention. 

In one specific prefen-ed embodiment, a system includes a Learn 
Module, a Review Module and a Test Module, each of which is arranged to 
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interact and adapt based on the performance and user results in the other two 
modules and the particular module itself. That is, operation and functioning of 
each of the Learn, Review and Test Modules are preferably changed in 
accordance with how a user performed in all modules. The Learn Module, the 
Review Module and the Test Module preferably define a main engine of the 
system which enables information to be encoded, stored 
and retrieved with maximum efficiency and effectiveness. 

A Discriminator Module may be included in the main engine to assist 
with the learning, reviewihg and testing of confusable items. 

A Schedule Module may also be included in the main engine to 
schedule the timing of operation of each of the Learn, Review and Test 
Modules. The scheduling is preferably based on a user's performance on 
each of the Learn, Review and Test modules, in addition to input infomiation. 
The Schedule Module completely eliminates all scheduling planning and tasks 
which are normally the responsibility of the user, and thereby greatly Increases 
the cognitive workload and metacognitive skills that the user can devote 
completely to learning, reviewing and testing of knowledge or skills. 

Further, a Progress Module may be included in the main engine for 
monitoring a user's performance on each of the Learn, Review and Test 
20 Modules so as to provide input to the system and feedback to the user 

whenever desired. The Progress Module presents critical information to the 
user about the processes of learning, reviewing and testing in such a manner 
as to enable the user to Increase his rhetacognitlve skills and become a much 
better learner both vinth the system of preferred embodiments of the present 
25 invention and also outside of the system. 

Also, a Help Module may be provided to allow a user to obtain further 
instructions and information about how the system works and each of the 
modules and functions thereof. The Help Module may include a help assistant 
that interactively detemiines when a user is having problems and provides 
information and assistance to overcome such difficulty and make the system 
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easier to use. The Help Module may provide visual, graphical, kinesthetic or 
other types of help information. 

It should be noted that although in the preferred embodiment of the 
present invention described in the preceding paragraph, the system preferably 
includes an interactive combination of Learn, Reviev^ and Test Modules, each 
module can be operated independently, and each module has unique and 
novel features, described below, which are independent of the novel 
combination of elements and the Interactive and adaptive operation of the main 
engine described above. 

In addition, other modules may be provided and used with the system 
described above. These other modules are preferably not included as part of 
the main engine, but instead are preferably arranged to interact with the main 
engine or various modules therein; For example, a Create Module may be 
provided outside of but operatively connected to the main engine to allow for 
input of knowledge or skills to be learned, retained or retrieved. The Create 
Module thus enables a user, administrator or other party to input, organize, 
modify and manage items to be learned so as to create customized lessons. 

An Input Module may also be included and arranged similar to the 
Create Module. The Input Module is preferably arranged, to allow a user, 
administrator, or other party to input any information that may affect operation 
of the modules of the main engine. Such input information may include 
infomiation about which of the main engine modules is desired to be activated, 
changes in scheduling of learning, reviewing or testing, real worid feed back ' 
which affect the learning, reviewing and testing and any other information that 
25 is relevant to the overall operation of the system and the modules contained in 
the main engine. 

Also, a Connect Module may be provided outside of but operatively 
connected to the main engine to all external systems such as computers, the 
Internet, personal digital assistants, cellular telephones, and other 
30 communication or infomiation transmission apparatuses, to be connected to 
the main engine. In fact, the Connect Module may be used for a variety of 
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purposes including allowing any source of information to be input to the main 
engine, allowing multiple users to use the system and main engine at the same 
time, allowing a plurality of systems or main engines to be connected to each 
other so that systems can communicate. Other suitable connections may also 
5 be achieved via the connect module. 

Another preferred embodiment of the present invention provides a 
method of learning including the steps of presenting knowledge or skills to be 
learned so that the knowledge or skills to be learned become learned 
knowledge or skills; presenting the learned knowledge or skills for review in a 
way that is different from the way in which the knowledge or skills are 
presented during learning, and presentirtg knowledge or skills for reviewing or 
testing whether the learned knowledge or skills have actually 
been learned. The method includes a step of monitoring each of the above 
steps and changing scheduling of each step based on progress in each step 
without the user knowing that monitoring or scheduling changes are occurring. 

As noted above, with respect to the Input Module, the main engine and 
the methods performed thereby, can communicate with the real world allowing 
for feedback, information exchange and modification of the operation of the 
modules of the main engine based on real world infomiation. All of these 
modules are preferably Interactive with the Schedule Module and scheduling, 
process which detemnines sequence of operation of the three modules and 
responds to the input information from the various input sources and optimizes 
the schedule of operation of the learn, review, and test processes. 

The system, including the various methods and apparatuses of 
preferred embodiments of the present invention, is constructed to have a 
highly adaptive interface that makes the system extremely streamlined and 
progressively easier to use each time a user operates any of the modules of 
the system. The system preferably prompts a new user for identification 
information such as a password or other textual, graphical, physiological or 
other identifying data that identifies each user. Then, the adaptive interface 
determines the pattern of usage, and with what level of skill that particular user 
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has operated the system. Based on this information, the system adapts to the 
user's familiarity level with the system and changes the presentation of 
infonnation to the user to make it easier and quicker to use the system. For 
example, cues, instmctions. help messages and other steps may be skipped if 
5 a particular user has operated the system many times successfully. 

Preferably, the Help module is preferably available should an advanced user 
forget how to operate the system. 

The various systems, methods and apparatuses of preferred 
embodiments of the present invention may take various forms including a 
0 signal earner wave format to be used on an Internet-based system, computer 
software or machine-executable or computer-executable code for operation on 
a processor-based system such as a computer, a telephone, a personal digital 
assistant or other information transmission device. Also, the systems, methods 
and apparatuses of prefen-ed embodiments of the present invention may be 
5 applied to non-processor based systems which include but 

are not limited audio tapes, video tapes, paper-based systems including 
calendars, books, and any other documents. 

The items to be learned, reviewed and tested using the systems, 
methods and apparatuses of preferred embodiments of the present invention 
0 are not limited. That is, items to be learned, reviewed and tested can be any 
knowledge, skill, or other item of information or training element which is 
desired to be learned initially and retrieved at a later date, or used to improve 
or build a knowledge base or skill base, to change behavior or thought 
processes, and to increase the ability to learn, review and test other items. For 
example, the systems, methods and apparatuses of preferred embodiments of 
the present invention may be used for all types of educational teaching and 
instajction, test preparation for educational institutions and various 
certifications such as CPA, bar exams, etc., corporate training, military and 
armed forces training, training of police offices and fire/rescue personnel, 
advertising and creating consumer preferences and purchasing patterns, 
mastering languages, learning to play musical instruments, learning to type. 
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and any other applications involving various knowledge or skills. That is, the 
real-v/orld applications of the systems, methods, and apparatuses of preferred 
embodiments of the present invention are not limited in any sense. 

Other features, advantages, elements and modifications of prefen-ed 
embodiments of the present invention will become more apparent from the 
detailed description of the present Invention below. 

BRIEF DESC RIPTION OF THE DRAWINirt.*^ 
A more complete appreciation of the present Invention and many of the 
attendant advantages thereof will be readily obtained as the same becomes 
better understood by reference to the following detailed description of 
preferred embodiments when considered in connection with the accompanying 
drawings, wherein: 

Fig. 1 is a schematic view of a system for learning, reviewing and 
testing knowledge or skills according to a preferred embodiment of the 
present invention; 

Fig. 2 is a graph of memory conditioning versus the CS-US interval 
related to preferred embodiments of the present invention; 

Fig. 3 Is a graph showing memory strength versus time indicative of the 
forgetting/retention function related to preferred embodiments of the present 
Invention; 

Fig. 4 Is a graph of memory strength versus time showing an expanded 
rehearsal series used to maintain a desired level of retention In the system 
shown in Fig. 1; 

Fig. 5 Is graph of frequency versus memory strength indicative of the 
signal detection theory with multiple distractors related to preferred 
embodiments of the present Invention; 

Fig. 6 is a matrix Indicative of the signal detection theory shown 
graphically in Fig. 5; 

Fig. 7 is a flowchart showing operation of a prefen-ed embodiment of the 
Learn Module of the system of Fig. 1 ; 
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Figs. 8A and 8B are flowcharts showing a Quick Review operation of a 
preferred embodiment of the Learn Module of the system of Fig. 1; 

Fig. 9 is a flowchart showing operation of a preferred embodiment of the 
Review Module of the system of Fig, 1 ; 

Figs. 10A and 10B are flowcharts showing operation of a preferred 
embodiment of the Test Module of the system of Fig. 1 ; 

Fig. 1 1 is a flowchart showing operation of a preferred embodiment of 
the Schedule Module of the system of Fig. 1 ; 

Fig. 12 is a flowchart showing operation of a prefen-ed embodiment of 
0 the Discriminator Module of the system of Fig. 1 ; 

Figs. 13A and 13B are flowcharts showing further operation of a 
preferred embodiment of the Discriminator Module of the system of Fig. 1 ; 

Fig. 14 is a graph of memory strength versus time indicative of the 
various levels of learning which can be achieved using the system shown in 
Fig. 1; 

Fig, 15 is a graph of the memory strength versus time which is indicative 
of the benefits of overlearning used in the system shown in Fig. 1; 

Fig. 16 is a table showing a learn presentation sequence in which cues 
and responses are presented in a certain sequence in the system of Fig. 1; 

Fig. 17 is a table showing a learn presentation pattern indicative of the 
order of presenting items to be learned as shown in Fig. 16; 

Fig. 18 is a table illustrating learn presentation timing indicative of the 
timing of the presentation of the items shown in Figs. 16 and 17; 

Fig. 19 is a graph of a the probability of recall according to the serial 
input position indicative of the serial position effect used in the system 
shown in Fig. 1 ; 

Fig. 20 is a graph of the mean number of rehearsals as a function of the 
serial input position used in the system shown in Fig. 1; 

Fig. 21 is a graph of memory comparison time versus memory span 
used in the system of Fig. 1 ; 
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Fig. 22 is a table showing a modality pairing matrix including various 
combinations of cues and responses used in the system of Fig. 1; 

Fig. 23 is a Review Curve Table which models curves indicative of the 
forgetting rate for each item learned in the system of Fig. 1; 

Fig. 24 is a Review Hopping Table which is a set of instructions for 
informing the system of Fig. 1 how to switch between review curves for each 
item to be reviewed; 

Fig. 25 is a graph of memory strength versus time which includes a 
family of review curves for illustrating hopping between review curves; 

Fig. 26 is a table showing various combinations of cues and responses 
showing the fomis for discrimination of two items used in the system of Fig. 1 ; 

Fig. 27 is a graph of latency of response versus the number of trials 
used in the system of Fig. 1; 

Fig. 28 is a graph of worl<load versus time indicative of schedule zones 
and workload used in the system of Fig. 1; 

Fig. 29 is an illustration of a main window display for a preferred 
embodiment of the system shown in Fig. 1; 

Fig. 30 is an illustration of a preview window display for a preferred 
embodiment of the system shown in Fig. 1; 

Fig. 31 is an illustration of a learn sequence including the presentation 
of a cue for a prefen-ed embodiment of the system shown in Fig. 1 ; 

Fig. 32 is an illustration of a learn sequence including the presentation 
of a cue and response for a preferred embodiment of the system shown in Fig. 
1; 

Fig. 33 is an illustration of a learn sequence including a request for 
faster or slower presentation of cues for a preferred embodiment of the system 
shown in Fig. 1 ; 

Fig. 34 is an illustration of a learn sequence including a completion 
indication for a preferred embodiment of the system shown in Fig. 1; 

Fig. 35 is an illustration of a learn sequence including a new item learn 
prompt for a prefen^ed embodiment of the system shown in Fig. 1; 
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Fig. 36 is an illustration of a learn sequence indicating a Quick Review 
operation for a preferred embodiment of the present invention; 

Fig. 37 is an illustration of a main window display with a review 
notification for a preferred embodiment of the present invention; 

Fig. 38 is an illustration of a review sequence including a presentation 
of review options for a preferred embodiment of the present invention; 

Fig. 39 is an illustration of a review sequence including an indication of 
an end of a review round for a preferred embodiment of the present invention; 

Fig. 40 is an illustration of a review sequence including a presentation 
of a cue to be reviewed for a prefen^ed embodiment of the present invention; 

Fig. 41 is an illustration of a review sequence including a user rating the 
quality of response for a preferred embodiment of the present invention; 

Fig. 42 is an illustration of a test sequence including a five alternative 
forced-choice for a preferred embodiment of the present invention; 

Fig. 43 is an illustration of a test sequence including a presentation of a 
cue and a feeling of knowing rating for a preferred embodiment of the present 
invention; 

Fig. 44 is an illustration of a test sequence including a cue and correct 
response for a preferred embodiment of the present invention; 

Fig. 45 is an illustration of a test sequence including scores of 
performance in the test sequence for a preferred embodiment of the present 
invention; 

Fig. 46 is an illustration of a schedule main window display for a 
preferred embodiment of the present invention; 

Fig. 47 is an illustration of a connect main window display for a 
preferred embodiment of the present invention; 

Fig. 48 is an illustration of a create control window display for a 
preferred embodiment of the present Invention; 

Fig. 49 is an illustration of a create main window display for a prefen-ed 
embodiment of the present invention; 
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Fig. 50 is an illustration of a progress main window display for a 
preferred embodiment of the present invention; 

Fig. 51 is an illustration of a help main window display for a preferred 
embodiment of the present invention. 

Fig. 52 is a schematic illustration of a preferred embodiment of the 
present invention in which the system of Fig. 1 is applied to a paper-based 
system; and 

Fig. 53 is an illustration of a review expansion series for the paper- 
based embodiment shown in Fig. 52. 

DETAILED DESCRIPTIOM QF PREFERRED EMROniMFMTg 

Hereinbelow, a plurality of preferred embodiments of the present 
invention are explained referring to the several drawings. Hereinafter, like 
reference numerals indicate identical or con-esponding elements throughout 
several views. 

Fig. 1 shows in a schematic form a system 10 according to a preferred 
embodiment of the present invention. The system 10 is arranged and 
operative to maximize the effectiveness and efficiency of learning, retaining 
and retrieving knowledge and skills. Knowledge in this system 10 preferably 
refers to declarative knowledge such as the knowledge of factual information. 
Skills in this system 10 preferably refers to procedural knowledge such as the 
knowledge of how to perfonm a task. Of course, other types of knowledge can 
be readily adapted for use In the system 1 0. 

The system .10 preferably includes a main engine 20. The main engine 
20 preferably includes a Learn Module 21, a Review Module 22 and a Test 
Module 23. The Learn Module 21 is adapted to encode knowledge or skills via 
a process for creating a memory record. The Review Module 22 is adapted to 
store knowledge or skills via a process of rhaintaining a memory record over 
time through rehearsal. The Test Module 23 is adapted to retrieve knowledge 
or skills via a process of producing a response to a presented cue 
automatically or through active recall. 
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The Learn Module 21 , the Review Module 22 and the Test Module 23 
preferably operate together and interact with each other to improve the 
learning, memory and performance of a user of the system 10. To this end, the 
cooperation between the Learn Module 21, the Review Module 22 and the 
Test Module' 23 allows a user to learn via a process by which relatively 
permanent changes occur in the behavioral potential as a result of interaction 
of these modules, to adiieve memory for each item which is the relatively 
permanent record of the experience that underlies the learning, and to achieve 
high levels of performance including various observable qualities of learning. 

As shown in Fig. 1, the Learn Module 21, the Review Module 22 and 
the Test Module 23 are preferably interactive with each other as shown by the 
an'ows connecting adjacent ones of the modules 21-23. As will be described 
in more detail below, the three modules 21-23 are preferably arranged such 
that the future operation of each of the modules 21-23 is based on the past 
performance in each of the other modules. 

The system 10 and the methods thereof can be implemented on any 
platform and with any type of system including a paper-based system, a 
computer-based system, a human-based system, and on any system that 
presents information to a person or organism, for learning and future retrieval 
of that information. For example, the system 10 may be a non-processor 
based system Including but not limited to audio tapes, video tapes, paper- 
based systems such as a word-a-day calendar described later with respect to 
Figs. 52 and 53, learning books such as workbooks, a processor-based 
system, such as that shown in Figs. 29-51, in which the main engine 20 is 
implemented in a processor, microprocessor, central processing unit (CPU), or 
other isystem in which functions are executed via processing of machine 
readable code, computer software, computer executable code or a signal 
carrier wave transmitted via the Internet. 

As will be described in more detail below, the main engine 20 may also 
include a Schedule Module 25, a Progress Module 26. a Help Module 27 and a 
Discriminator Module 28. 
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The system 10 may also be adapted to interact with various elements 
or modules external to the system, such as an Input Module 100. a Create 
Module 200 and a Connect Module 300 shown in Fig. 1 . 

It should be noted that the modules 21, 22, 23, 25, 26, 27, 28, 100, 

5 200. 300 and other modules described herein are preferably processes or 
algorithms including a sequential series of steps to be performed. The steps 
may be performed via a plurality of different devices, apparatuses or systems. 
For example, the steps to be performed by the main engine 20 including 
modules 21 . 22, 23. 25. 26. 27. 28 may be perfomied by various devices 

0 including as a computer, any type of processor, a central processing unit 

(CPU), a personal digital assistant, a hand-held electronic device, a telephone 
including a cellular telephone, digital data/information transmission device or 
other device which performs the steps via processing of instructions embodied 
in machine readable code or computer executable code such as computer 

5 software. 

Each of the modules external to the main engine 20 will be described 
and then each of the modules of the main engine 20 will be described. 

The processes or steps to be performed by the Input Module 100, the 
Create Module 200 and the Connect Module 300 may be performed by various 

0 devices including a keyboard, microphone, mouse, touchscreen, musical 
keyboard or other musical instrument, telephone, Internet or other suitable 
information transmitting device. 

The Input Module 100 may be adapted to receive information that is 
transmitted overtly or covertly from the user. The Input Module 100 can also 

5 be used by an administrator of the system such as a teacher. The Input 
Module 100 can also receive input information from objects or any other 
source of information existing in the real world. The Input Module 100 is 
configured to allow a user, administrator, or other party or source of 
information to input any information that may affect operation of the 

) modules of the main engine 20 or other modules in the system 10. Such input 
information may include information about which of the main engine modules is 
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desired to be operated, changes in scheduling of learning, reviewing or testing, 
user performance with the system 1 0, the type and difficulty of the items to be 
learned, reviewed and tested, real world feed back which affect the learning, 
reviewing and testing, and any other information that is relevant to the overall 
5 operation of the system 10 and the modules contained in the main engine 20 
and outside of the main engine. 

Furthermore, the Input Module 100 can be configured to receive 
information as to the performance of the user of the system 10 through 
quantitative measurements such as time required to input various responses 
10 requested, ability of user to meet and adhere to schedule set up by the system 
10, and the user's level of interest and arousal in learning which can be 
measured by such physiological characteristics as perspiration, pupil diameter, 
respiration, and other physiological reactions. 

As will be described in more detail below, the system 10 accepts and 

15 obtains various input information through the Input Module 100 and the future 
operation of the various modules of the system 10 modified based on this input 
information. In this way, the system is adaptive to the user's abilities and 
performance, and other input information so as to constantly and continuously 
adapt to provide maximum effectiveness and efficiency of learning, retaining 

20 and retrieving of knowledge or sidlls. 

The manner in which the information is input to the system 1 0 via the 
Input Module 100 is not limited, and may include any information transmission 
methods, processes, apparatuses and systems. Examples of input devices 
and processes include electronic data transmission and interchange via 

25 computer processors, the Internet, optical scanning, auditory input, graphical 
input, kinesthetic input and other known information transmission methods and 
devices. 

The Create Module 200 may be provided outside of, but operatively 
connected to, the main engine 20 to allow for input of knowledge or skills to be 
30 learned, retained, and retrieved. The Create Module 200 thus enables a user, 
administrator or other party to create new customized lessons by inputting 
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items to be learned and by providing additional information about each item 
that will affect how each item or groups of items will be learned, reviewed and 
tested to maximize effectiveness and efficiency of the learning system 10. 

The Connect Module 300 may be provided outside of, but operatively 
connected to, the main engine 20 so as to connect all types of external 
systems and devices such as computers, the Internet, personal digital 
assistants, telephones, and other communication or information transmission 
apparatuses, to be connected to the main engine 20. In fact, the Connect 
Module 300 may be used for a variety of purposes including allowing any 
source of infomnation to be Input to the main engine, allowing multiple users to 
connect to and use the system 1 0 and the main engine 20 at the same time, 
and allowing a plurality of systems 10 or main engines 20 to be connected to 
each other so that systems 10 and main engines 20 can communicate and 
share infomrjation such as lessons to be learned, perfonnance by an individual 
in any of the modules, changes in schedule and many other factors, data and 
information pertaining to operation of the system 10. Other suitable 
connections may also be achieved via the Connect Module 300. 

The Help Module 27 may be provided to allow a user to obtain further 
instructions and information about how the system 10 works and the operation 
of each of the modules and functions thereof. The Help Module 27 may 
include a help assistant that interactively determines when a user is having 
problems in operating the system 10 and provides information and assistance 
to overcome such difficulty and make thei system 10 easier to use. The Help 
Module 27 may provide visual, graphical, kinesthetic or other types of help 
information to the user, either in response to a request from the user or when 
the system 10 has detected that the user is having difficulty using the system 
10. The Help Module 27 may also provide feedback, preferably through the 
Connect Module 300, to an administrator such as a teacher or some of other 
third party so as to indicate problems that various users of the system 10 are 
having. 
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The Progress Module 26, the Schedule Module 25 and the 
Discriminator Module 28 will be described after the Learn Module 21, the 
Review Module 22 and the Test Module 23 are described. 

The arrangement and operation of the system 10 of Fig. 1 and all of the 
5 elements thereof are based on several scientific principles and phenomena 
which are related to learning systems having memory with a fixed storage 
space for storing knowledge or skills. 

Fig. 2 is a graph of the degree of memory conditioning versus the CS- 
US Interval which is a known characteristic of temporal aspects of classical 
0 conditioning. Classical conditioning is the procedure in which an organism 
comes to display a conditioned response to a neutral conditioned stimulus that 
has been paired with a biologically significant unconditioned stimulus that 
evoked an unconditioned response. For example, in the well known 
experiment by Pavlov, a dog comes to display the conditioned response of 
salivating upon hearing a bell. In this example, the ringing of the bell is the 
neutral conditioned stimulus which is paired with the biologically 
significant unconditioned stimulus of presentation of food which causes the 
biological reaction of the dog salivating. 

Similar biological principles apply to operant conditioning or 
instrumental conditioning which is the procedure in which a particular stimulus 
condition occurs and if an organism voluntarily emits a response to the 
stimulus, then a particular reinforcer will occur. For example, a student wishes 
to learn that the Spanish word for dog is "perro." The stimulus can be thought 
of as "dog" and the response "perro", and the reinforcer may be the teacher's 
approval. 

As seen in Fig. 2, ideally, when information is initially encoded, or is 
strengthened through reviewing or testing, a cue Is presented and a response 
is actively recalled. It is this process of active recall that strengthens 
memories. One critical aspect of the process of achieving active recall of 
knowledge or skills is the timing of the presentation of the cue and the 
presentation of the response. Fig. 2 shows that maximum conditioning occurs 
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when the response follows the cue by about 250 milliseconds to about 750 
milliseconds. 

While classical conditioning and operant conditioning have been used 
in the past for various training and teaching methods, these methods have not 
been quantitatively measured and then had the resulting quantitative 
measurements used to modify various timing parameters and steps of a 
learning process as in preferred embodiments of the present invention as will 
be described later with respect to Figs. 7-13. One of the advantageous results 
of this novel process is maximizing the effectiveness and efficiency of 
encoding for retrieval such that a paired-associate or item to be learned is 
encoded to a level of automaticity and can be recalled automatically with no 
significant cognitive effort being expended! A real world example of this novel 
process is in the advertising context in which a paired-associate might include 
"sneaker" as the cue and "Brand X sneakers" as the response. With the novel 
process of preferred embodiments of the present invention, the system 
adaptiveiy and interactively encodes for retrieval such that when a consumer is 
presented, in any form, with the cue "sneaker", the consumer automatically 
thinks of the response "Brand X sneakers." 

As will be described below in preferred embodiments of the present 
invention, the presentation of cues and responses in the Learn Module 21, the 
Review Module 22 and the Test Module 23 interactively adapts the CS-US 
interval shown in Fig. 2 based on various factors such as the type and difficulty 
of the knowledge or skills, the user's performance in each of the modules of 
the system 10, the measured arousal and attention of the user, the measured 
confidence of the user in responding to the presentation of cues and 
responses and providing responses to cues, the number of times a paired- 
associated has been seen by a user to take Into account the effects of 
habituation and sensitization, the user's feeling of knowing and judgement of 
knowing as quantitatively rated by the user, the measured latency of response 
of the user, the measured memory strength for a particular item, the measured 
probability of recall and user's performance, and many other quantitatively 
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measured factors and effects. Fig. 3 shows a graph of memory strength 
versus time which indicates how human memory decays overtime which is an 
important phenomenon in a learning system, The graph of Fig. 3 is often 
referred to as the forgetting function because the vertical distance between the 
5 curve and the horizontal line marking the maximum memory strength 

represents the amount of previously learned material that has been forgotten. 
Conversely, the graph of Fig. 3 is also referred to as the retention function 
because the vertical distance between the curve and horizontal line marking 
the minimum memory strength represents the amount of previously learned 
10 material that has been retained or remembered. As seen in Fig, 3, the curve 
Is a negatively accelerated function which means that initially, material is 
forgotten quickly and over time, the rate at which material is forgotten slows. 
The curve shown in the graph of Fig. 3 is measured by a test of memory at a 
fixed degree of serisitivity. 

Figure 4 is a graph similar to Figure 3. The axes of Fig. 4 are memory 
strength and time as in Fig. 3. Starting from t=0. the trace proceeds quickly to 
a local maximum, indicating the desired degree of initial learning of previously 
unlearned material. Following an initial learning session, the trace declines in 
the form of a negatively accelerated function indicating the normal loss of 
20 memory strength over time. It is desirable, however, to maintain a certain level 
of retention for learned material over some period of time. 

Conventional methods of learning have recognized the effects of the 
decay of memory over time as shown In Figs. 3 and 4 and have used an 
expanded rehearsal series, whereby items previously learned are later 
25 reviewed according to a schedule which is not modified and Is identical for all 
Items and individuals. The expanded rehearsal series shown in Fig. 4 is a 
random and crude attempt at minimizing the effects of forgetting due to the 
decay of memory over time. 

In contrast to the conventional methods, preferred embodiments of the 
present invention quantitatively measure the memory strength for each item 
and for each user, since there are significant differences in memory strength 
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over time for various types of knowledge or skills and for various users which 
have vast differences In how they encode, store and retrieve knowledge or 
skills, (i.e. dyslexic, learning disabled, low IQ, etc.). The memory strength over 
time is quantitatively measured using overt and covert information gathered 
5 during the user's operation and activity in the Learn Module 21, 

the Review Module 22 and the Test Module 23, as well as other modules. The 
Information input to the system 10 to detennine memory strength over time 
include, but are not limited to: rate of initial learning, degree of initial learning, 
probability of recall, latentqr of recall and siavings in relearning. The 
0 quantitative measurement of the memory strength for each Item is used to 
adaptively modify the operation of one or more of the Learn Module 21 , the 
Review Module 22 and the Test Module 23, as well as other modules included 
in the system 10. 

More specifically, the system 10 determines that the memory strength 
for a particular item has decreased to the minimum retention level by making 
calculated projections based on the mathematical characteristics of the decline 
of human memory, the type and difficulty of the item being learned, the 
recency, the frequency, the pattern of prior exposure, and the user's particular 
history of past use of the system 10. As can be seen in Fig. 4, items seen 
twice are forgotten more slowly than items seen once and furthermore, items 
seen three times are forgotten more slowly than items seen twice or once. 
This must be taken into account when making the calculated projections as to 
when the memory strength for each particular item will fall below the minimum 
retention level. The system 1 0 schedules the item for review in the Review 
Module 22 based on the calculated projections. The climb of the trace of Fig. 
4 back to the local maximum memory strength indicates the change that occurs 
as a result of a review session in which a previously learned item is reviewed 
in the Review Module 22. Following the Review Module 22 session, the trace 
of memory strength of Fig. 4 declines once again in the fomi of a negatively 
accelerated function. This time, however, the curve function Is more shallow 
than the forgetting curve following the initial learning session. The more 
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shallow curve indicates that the item is forgotten more slowly for items seen 
twice than for items only seen once. Once again, when the system 10 has 
determined that the memory strength for each particular item has decreased to 
the minimum retention level, the system schedules the item for review. This 
5 process of forgetting and review continues for as long as the user or 

administrator desires the learned material to be retained to a desired level. 

Because all learners are not alike, and not all items are equally easy to 
learn, to maintain in memory and to retrieve, the system 10 preferably 
constantly monitors the memory sti^ength for each item for each learner to 

1 0 determine the most effective and efficient schedule of Review. 

Figure 4 illustrates the schedule of review for items that a model learner 
finds are of average difficulty. The small vertical hash marks above the curves 
in Fig. 4 indicate the end of each Review session. The spacing of the hash 
marks in Fig. 4 is indicative of an expanded rehearsal series. Above these 

15 hash marks are another series of hash marks which indicate the spacing of 
review sessions for items a model learner finds are easy to learn, to maintain 
in memory or to retrieve. Above these hash marks is a third set of hash marks 
which indicate the spacing of review sessions for items that are relatively 
difficult. 

20 In the system 1 0 of preferred embodiments of the present invention, 

there is no single review schedule in the Review Module 22 that is the most 
effective and efficient to maintain a desired level of retention for each user for 
each item. Accordingly, the system 10 monitors the users as he learns, 
reviews and tests himself on each item. Based on measured quantitative 

25 results gathered overtly and covertly as described above, the system 10 
quantitatively determines when the next review session must 
occur to maintain the desired level of retention. Thus, the system 10 is 
adapted to the individual needs of each user. 

Eventually, review sessions are scheduled so far apart in time, that the 

30 item can be considered to have entered a state of permastore. That is, the 
item will have been learned and reviewed such that the Item is known for the 
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lifetime of the learner. Although memory strength will not decay to a point 
where the item is lost due to storage failure, the item may be forgotten as a 
result of low memory activation and the user may experience a retrieval failure. 
This problem can be reduced or eliminated by scheduling review sessions for 
particularly important items to maintain a minimum desired level of activation! 

Figure 5 illustrates the concept of Signal Detection Theory, a branch of 
psychophysics. Signal Detection Theory is based on the phenomenon that a 
living organism such as a human or other animal, perceives stimuli and makes 
decisions based upon those perceptions. This two-part process is integral to 
many memory related tasks and is quantitatively incorporated into the 
performance of various modules of the system 10 of a prefen-ed embodiment of 
the present invention. 

In Fig. 5, a target, or correct response to a cue. is perceived as differing 
in memory strength from a number of distracters. The user's ability to perceive 
the difference between the target and the distracter{s) is measured by d', also 
known as performance. In the signal detection paradigm, the user must be 
able to discriminate the target from the distracters. The criterion a user uses in 
making decisions about signal existence is known as Beta. If the user is 
extremely lax in their criteria for reporting, Beta shifts to the left of the graph of 
Fig. 5. If the user is extremely cautious in their criteria for reporting, Beta shifts 
to the right of the graph of Fig. 5. 

Figure 6 is related to Fig. 5 and shows a signal detection theory matrix. 
When there is overlap between the target and the distracters, the position of 
Beta on the graph of Fig. 5 creates a possibility of four outcomes. In memory 
25 experiments where a user is trying to retrieve a correct response to a 

presented cue, the user must select a response stored in his memory from a 
number of alternative incorrect responses and distracters. Four outcomes are 
possible in the simplest case: In the first case, the user believes that he has 
retrieved a response that is correct, and he turns out to indeed be correct — a 
30 correct recognition. In the second case, the user believes that he has 
identified an incorrect response, and he is correct in hls assessment and 
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reporting — a correct rejection. In tlie tliird case, the user believes that he has 
identified a correct response and reports it as such. Unfortunately, the chosen 
response is incorrect — a false alarm. In the fourth case, the user believes 
that he has identified an incorrect response and reports it as such, but it turns 
5 out that it was actually the correct response — a false rejection. ' 

The system 10 according to preferred embodiments of the present 
invention monitors not only the correctness of the user's response but also the 
user's performance, which is the ability to evaluate accurately whether they 
know the correct response and the Incorrect responses. The system 10 
1 0 according to prefen-ed embodiments of the present Invention also measures 
the time required for the user to make such evaluation about the con-ect 
response and incorrect responses. 

Instead of using the measured performance to generate sequences of 
perceived known and unknown items which is not done in any of the preferred 
1 5 embodiments of the present invention, the quantitatively measured 

performance is fed back and presented, either graphically, auditorily, " 
kinesthetically, or otherwise, to the user, preferably along with the score of 
accuracy of recall, to provide information to the user about his metacognitive 
skills in this learning environment and other learning environments, enabling 
20 the user to improve how he monitors and controls how he learns and to 

become a better learner. Because a significant part of learning and retrieval is 
the ability to discriminate between correct and incorrect answers, the system 
10 according to preferred embodiments of the present invention not only 
teaches the user knowledge or skills, but also trains the user to become a 
25 more effective learner by improving the metacognitive skills required for self- 
paced learning. These are skills necessary to monitor perfonnance during 
learning, reviewing, and testing. Metacognitive skills include subjective 
measurements of feeling of knowing, confidence, and judgement of learning, 
which are measured quantitatively in preferred embodiments of the present 
30 invention and then used to modify the future use of the system 1 0 and the 
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future operation of the various modules therein, including especially the Learn 
Module 21, the Review Module 22 and the Test Module 23. 

For example, the system 10 of preferred embodiments of the present 
invention preferably uses the measured probability of recall, latency of 
5 response, and savings in relearning in the future operation of the Learn 
Module, the Review Module and the Test Module to further increase the 
effectiveness and efficiency of learning and performance achieved by the 
system of the present invention. 

Each of the modules including the Learn Module 21 , the Review 
10 Module 22 and the Test Module 23 are preferably arranged and adapted to 
function either together with the other two of these three modules or to function 
independently as a stand-alone module. 

In addition, other modules such as the Schedule Module 25, the 
Progress Module 26 and the Help Module 27 may be added to any ' 
1 5 combination of the Learn Module 21 . the Review Module 22 or the Test 
Module 23 in a system. 

It should be noted that each of the Learn Module 21 . the Review 
Module 22 and the Test Module 23 contain many novel aspects, processes, 
elements and features thereof and can be used independently of the system 
shown in Fig. 1 and independently of the other modules of the main engine 20 
and the system 10 shown in Fig. 1. The novel features of each of the Learn 
Module 21, the Review Module 22 and the Test Module 23 will be described 
now. 

Now. each of the various modules will be described. 
I. THE LEARN MODULE: The Learn or Encode Module 21 Is used to present 
Items to be learned to a user. Learning methods have been known such as the 
Skinner method described above. The method and system according to the 
preferred embodiments of the present Invention is based on the Skinner 
method but is modified and greatly improved so as to be adaptive and 
30 interactive in response to various factors. 



20 



25 



wo 01/50439 



PCTAJSOO/35381 



33 

The Learn Module 21 uses the Skinner method of (earning through 
presenting paired-associates of cues and responses. The timing, order of 
presentation and sequence of each cue and response for the Learn Module 21 
is interactively determined based on covert and overt input from the user and 
may also may be based on information received from various other input 
sources. Such covert or overt input may relate to the content of knowledge or 
skills to be learned, timing of presentation of knowledge or skills to be learned 
including timing between each cue and response in each of the plurality of cue 
and response items, timing between presentation of groups 
of cue and response items (time between presentation of one cue and 
response pair and the next cue and response pair), sequence of presentation 
of knowledge or skills to be learned and the format of presentation of 
knowledge or skills to be learned and other factors. 

The inputs upon which the presentation of the items in the Learn 
Module 21 may be from one or more of the user, the administrator, the system 
10 Including other modules included therein, and any other input source that is 
relevant to the learning process and operation of the Learn Module 21. 

For example, other input sources could be sensed environmental 
conditions such as time of day. Time of day has an effect on learning for 
people of various ages and therefore may be input to change the presentation 
of items in the Learn Module 21. The inputs to the Learn Module 21 may 
further include various personal and physiological information such as age, 
gender, physiological activity such as galvanic skin response, Information 
obtained through non-invasive monitoring of brain activity, and other personal 
factors. 

The overt and covert inputs from the various input sources may include 
information concerning rate of presentation of items, format of presentation of 
items, sequence of presentation of items, and other information that would 
affect operation of the Learn Module 21 . The method of inputting the overt 
information is based on a purposeful, conscious decision on the part of a user, 
administrator, or other source to input information to the system. In contrast, 
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the covert information is input based on physiological information obtained by 
various sensors obtaining data regarding factors such as a galvanic skin 
response, pupil diameter, respiration, blood pressure, heart rate, brain activity, 
and other personal conditions. This information can be obtained by such 
known sensors including an electromyogram. electroencephalogram, 
thermometer, and electrocardiogram among others. The covert information is 
analyzed to determine many factors including a user's attention and vigilance 
so as to determine to what degree a user is attending to the presentation of 
Information in the Learn Module 21 . 

The actual cue and response items may be modified in format according 
to the desires of a user, administrator or based on other input information. In 
addition, the cue and response items may be supplemented by information 
such as a facility for pronunciation hints, and other helpful facts or InformaUon 
related to the items being presented for learning in the Learn Module 21 : Such 
15 additional related information is 

not part of the cue and response items but is presented with the cue and 
response items to assist in the learning process. 

In addition, items to be learned in the Learn Module 21 may be 
confusable items and may be presented differently from other items to be 
learned. This process will be described in more detail in the description of the 
Discriminator Module 28 below. 

The Learn Module 21 operates and is controlled based on many factors 
including desired degree of initial learning and desired degree of retention 
over time. A desired degree of initial learning may be input by a user, 
administrator, or other input source to indicate what degree of rtiemoiy strength 
is desired for each item or group of items to be learned. The desired degree of 
retention is based on the rate of forgetting predicted (Figs. 3 and 4) and 
measured by probability of recall, latency of recall, savings in releaming, and 
other factors, in Review and Test sessions conducted over time. 

As a general rule, the system, apparatus, and method of prefen-ed 
embodiments of the present invention seek to provide a level of retrieval that is 
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known as automaticity. Automaticity means that a pereon knows the 
knowledge or skills and does not have to expend great effort to remember it 
Automaticity decreases the latency of response as well as the cognitive 
workload during retrieval. 

Therefore, the preferred embodiments of the present invention perform 
encoding for automaticity to achieve "knowing rather remembering." The prior 
art assumes mastery Is achieved at the time that the first correct answer is 
provided on a test of recall. Recall, however, is not automaticity. -Automaticity 
can be distinguished from recall because it allows extremely fast retrieval of 
knowledge or skills. The difference between automaticity and recall is latency 
of response or how long it takes to respond 

to a cue or perfomn a desired skill. Also, simple recall requires relatively-more 
cognitive effort on the part of the person responding to the cue or performing 
the skill, but automaticity requires far less cognitive effort thereby reducing 
overall cognitive workload. The net result is that knowledge or skills encoded, 
retained and retrieved using the method are retrieved quickly and effortlessly. 

In order to reduce cognitive workload during learning and thereby 
reduce fatigue, the pattern, sequence and timing of presentation of items is 
continuously adjusted in the Learn Module 21 based on quantitative inputs 
thereto. Items to be (earned are preferably presented one item at a time to 
avoid requiring the user to retain multiple items in short term memory. In 
addition, the pattern, sequence and timing of items to be learned is determined 
by the system 10 and therefore, the cognitive effort required 
for monitoring and controlling the study session is reduced so that a person 
can leam for a longer period of time and Is not distracted from the learning 
process. 

The Leam Module 21 also operates to capture and maintain the user's 
attention based on psychological phenomenon of habituation and 
sensitization. One example of sensitization is a person becoming aware of 
something, such as the sound of a car alami, which initially captures that 
person's attention. However, if that stimuli repeats itself over and over, the 
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person becomes oblivious or habituated to it - their brain tunes it out. 
Accordingly, the presentation pattern, sequence, or timing of items to be 
learned may be preferably varied so as to vary stimulation in such a way as to 
avoid habituation or the disengagement from attending to this particular stimuli. 

5 The difficulty is that this variation in the above-identified factors should not be 
done overtly but should be done in a manner that is not so obvious that it 
becomes a distraction in and of itself, but rather should be done in a more 
subtle manner, using variations In the presentation pattern, sequence, and 
timing at the "just noticeable difference' threshold whereby a person notices 

0 unconsciously but not actually consciously, In a preferred embodiment of the 
Learn Module 21, attention that is declining is recaptured through various 
means such as the use of obligatory attention cues and then by varying the 
presentation pattern, sequence, and timing of the items presented. 

Obligatory attention cues include such sensory events as a blinking 

5 light, a tone, object movement or other stimulation that attracts the attention of 
the user. 

In addition, the serial position affect is preferably taken into 
consideration in the Learn Module 21 and the presentation of items to be 
learned in the Learn Module 21 is changed in order to eliminate the serial 
3 position effect. Providing a non-serial presentation to avoid the serial position 
effect may be accomplished by reordering the presentation of the cue and 
response items. For example, the non-seriail presentation 
of items in the Learn Module 21 can be achieved by spacing apart unknown 
items to be learned by inserting between the unknown items, a number of 
items which are randomly selected from a pool of previously learned items. 

The Learn Module 21 takes advantage of the psychological 
phenomenon known as the spacing effect. The spacing effect states that for an 
equal number of presentations of an item to be learned, distributing the 
presentations overtime yields significantly greater long-term retention than 
does massed presentations. Furthermore, the spacing of presentations in the 
Learn Module 21 preferably takes the form of an expanded rehearsal series 
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Where items are reviewed at increasingly longer intervals for the greatest 
effectiveness and efficiency of learning. 

Also, the sequence in which the cue and response items are presented 
in the Learn Module 21 may be changed to present more difficult items more 
times than easier items allowing the user to concentrate their effort where it is 
most needed. 

The Learn Module 21 is preferably designed to promote self-motivated 
learning. One factor in motivating learning is the rate of success and failure. 
Too much success or failure is not motivating to a person seeking to learn. 
Thus, the Learn Module 21 maintains a challenging learning environment by 
sequencing the presentation of paired-associate items to balance items that a 
user is successful at providing a correct response to with items the user is less 
successful at providing a correct response to. 

The Learn Module 21 also tiakes into consideration a physiological 
phenomenon known as consolidation in the presentation of items to be 
learned. Consolidation is the period of time immediately following learning 
where memories are most vulnerable to loss due to decay and interference. In 
first stage of memory formation, process oriented changes take place at the 
cellular level of the brain resulting in short term memory. During consolidation, 
additional changes occur and result in actual structural modifications in the 
brain. This is prerequisite for long term memory formation. Taking this into 
consideration, the Learn Module 21 presents items as many times as is 
necessary to achieve the desired degree of overleaming. In contrast, in the 
prior art, learning is judged to be completed when the user is able to recall the 
correct response to a cue the first time. 

Overleaming suggests that the user can derive additional benefit from 
continuing to study an Item learned to this level. One measure of overieaming 
is latency of recall. An item that is overlearned will be recalled not only 
dorrectly, but also quickly, indicating automaticity. Overleaming, however, is 
subject to the law of diminishing returns, which means that at some point the 
effort expended does not provide a justifiable benefit. Overleaming In the 
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Learn Module 21 reduces the likelihood that memories will be lost during 
consolidation and that if no review were to follow, the likelihood of successful 
retrieval at some future date would be higher than if the items were not 
overlearned, as shown in Fig. 15, which will be described in more detail below. 

5 As will be described in more detail below, all of the modules of the main 

engine 20 are preferably adapted to enable users to become better learners by 
training them to make more accurate metacognitive judgements. Judgment of 
learning, fpr instance, is a subjective evaluation made after a learning session 
in which a person judges whether an item was learned or not learned. In self- 

0 paced study, the decision as to whether to continue studying a particular item 
is often made based on the user's judgement of learning of that item. An 
inaccurate judgement will lead to either too much time, or too 
little time spent on an item resulting in less effective and efficient learning than 
would othenyise be possible if an accurate judgement were made. 

5 In addition, in the Learn Module 21 , it is preferable to provide a preview 

of knowledge or skills to be learned. In the preview, a background description 
or related information is provided before the actual cue and response items to 
be learned are presented. Such background information can include general 
information about a topic that is the subject of the cue and response items so 

0 as to provide some basis or context for learning, or what a user should keep in 
mind while learning, hints about the 

upcoming lesson itself or any other relevant information. The preview 

information can be text-based, graphics-based, auditory, or any other format. 

In addition, the preview can teach a user how to learn more effectively and 
5 efficiently before he learns, for example, by providing learning tools 

(pronunciation hints, study tips, what to pay attention to, etc.). 

The Learn Module 21 preferably includes a Quick Review, which is 

presented at the end of lesson. Quick Review provides the user one or more 

opportunities to review difficult or unlearned items before that particular 
3 session of the Learn Module 21 is completed. Quick Review preferably 

reorders the presentation of items so as to eliminate the serial position effects 
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Of primary, recency. Von Restorff and other well known effects. In addition, it 
is possible in Quick Review to rearrange the cue and response items such that 
for each item, the cue becomes response and the response becomes the cue. 

Preferably, items presented during Quick Review are sorted using the 
drop-out method. That is, if the user quickly indicates he is able to retrieve a 
correct response to presented cue, as measured by accuracy of recall and 
latency of response, the item is dropped out of the list of items being presented 
because the item is determined to be. well known. The remaining items are 
then re-ordered and lesser knovifli items are presented again. This continues 
until no items remain, or until some other critiBria is met such as the completion 
of four rounds of Quick Review. 

The re-ordering done during Quick Review is preferably based on an 
inside-out ordering to reduce the serial position effect. Primacy and recency 
effects cause items presented first and last to be learned better than Items in 
the middle of the sequence. By turning the sequence inside-out in terms of 
presentation, the effects of primacy and recency are minimized ensuring that 
items originally presented in the middle of the sequence are learned to the 
level of items originally presented at the beginning or end of the sequence. 

In the Learn Module 21. the ease of initial learning of each item can be 
determined by analyzing the drop-out scores. This is done by measuring how 
many times an item was presented and determining from this the relative 
difficulty of learning each item. This information is then used to place the item 
on the appropriate review curve (described later) which determines the initial 
schedule of review. 

The Learn Module 21 is preferably interactive with the Review Module 
22 and Test Module 23. More specifically, the ease of initial learning in the 
Learn Module 21 as described above is used to determine how to present 
items in the Review Module 22. 

More specifically, preferred embodiments of the present invention use 
hopping tables, prediction curves and other mathematical correlations to 
accurately control interaction between the modules of the main engine 20. For 



wo 01/50439 



PCT/USOO/35381 



40 

example, a Learn hopping table is preferably provided and used to determine 
the initial schedule of presentation of items in the Review Module 22. Using 
the Learn hopping table, if an item was presented once in Quick Review. It is 
placed on an easy cun/e— one which schedules the review relatively 
5 infrequently. If an item was presented twice in Quick Review, it is placed on a 
medium curve. If an item was presented three times in Quick Review, it is 
placed on a hard cun/e— one which schedules review frequently. As will be 
described below, as a user begins reviewing the learned items using the 
Review Module 22. items will hop from curve to cun/e; the curves determine 
the items to present during each review session of the Review Module 22. The 
Review Module 22 has a hopping detect function which feeds back Into a rule 
set used to determine which review curve the item is on and Is used to 
reconfigure the hopping table rules in the Learn Module 21 for improving the 
effectiveness and efficiency of learning, reviewing, and testing in the future. 

Since human memory decays differently for words, pictures, sounds, 
smells, skills and other types of information and depends on degree of difficulty 
in learning, retaining, and retrieving items, it is preferable to modify the curves 
to reduce hopping to more efficiently predict the decline of memory strength as 
a result of decay and interference. A plurality of families of curves may be 
used and arranged according to the characteristics of the curves. Such 
hopping tables and families of curves for revievv are shown in Figs. 24 and 25 
and will be described later, there are preferably several sets of hopping njles 
assodated with each curve. The system 1 0 determines how 
many times an item has hopped between curves and will determine which 
cun/e included In which curve family minimizes hopping because too much 
hopping indicates poor prediction of decline of memory strength for that user 
and that item. 

The function of the system 1 0 relating to the hopping rules and curves 
depends on the rate or level of retention chosen by the user or administrator. 
Different families of curves may be better at predicting items based on primary 
sensory modality or other factors. Also, the cun/es or families of curves may 
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be chosen for use based on subject matter of content, gender, age, or each 
individual user since information about each user may be made available each 
time the system starts up. This information about 

how the user learns is then used by the system in each of the Learn Module 
5 21 , the Review Module 22, and Test Module 23. 

With such data, it is possible to determine which items are more difficult 
to learn, retain and retrieve than other items based on data from many other 
users and to share that data with each specific user so as to affect how the 
Learn Module 21, the Review Module 22 and the Test Module 23 perform for 
10 that user. 

H. THE REVIEW/STORE MODULE: The Review Module 22 preferably 
includes many different types of review fohnats including Norma! Review, Ad 
Hoc Review, and Scheduled Review. 

In a preferred method of the Normal Review of the Review Module 22, 
after a lesson has been presented by the Learn Module 21 and has been 
learned by a user, the system 1 0 prompts the user to indicate whether the 
lesson is to be reviewed in the future. If so, the system 1 0 places the lesson 
on a review schedule of the Review Module 22 for maintaining default 
retention rate for an Indefinite period of time. If not, then no review schedule 
is created for that lesson. The user or administrator may change from "never 
reviewed" to "indefinite review," or vice versa, at any time in the future. The . 
user or administrator may also change the retention level from the default level 
to any other level at any time in the future for lessons or individual items. 

The Schedule Module 25 schedules the appropriate time of learning, 
reviewing and testing of items based on a previously Input desired date of ' 
completion as well as many other factors. The desired date of completion is 
the date by which the user desires all of the items to be known to a 
predetermined level of memory strength and activation, preferably to a level of 
30 automaticity. At the appropriate times, the system will indicate that the items 
scheduled for review are due to be reviewed and review 



15 



20 



25 



wo 01/50439 



PCT/USOO/35381 



42 

proceeds as will be described below in more detail. 

Scheduled Review of the Schedule Module 22 takes into account 
problems such as items learned later in the schedule have relatively higher 
activation and relatively lower strength than items learned early in the 
5 schedule which have relatively higher strength and relatively lower activation. 
Items that are more difficult to learn may be scheduled to be learned early in 
the overall schedule to provide them with the greatest number of review 
sessions to develop the desired degree of memory strength. 

With Ad Hoc Review of the Review Module 22, a user can select a 

1 0 particular item or group of items to be reviewed at that moment If the user 
conducts this review on an ad hoc basis instead of waiting for the review of the 
item or group of items scheduled for Nonmal Review, feedback based on Ad 
Hoc Review performance is used by the system 1 0 to reschedule future 
Normal Review, Scheduled Review and testing of this item in the Test Module 

15 23. 

Scheduled Review of the Review Module 22 arranges the presentation 
of items to be reviewed so as to increase memory strength of items learned 
later in the schedule and increase memory activation of items learned early in 
the schedule just prior to the date when the knowledge or skills are required. 

20 Other factors which might be used by Scheduled Review of the Review 

Module 22 to an-ange presentation of items for review may include degree of 
difficulty, degree of importance, strength, activation and how user has 
interacted with the system 1 0 in the past. 

In addition. Scheduled Review and Normal Review of the Review 

25 Module 22 preferably take into account graceful degradation and workload 
smoothing when arranging the presentation of items to be reviewed. Graceful 
degradation and workload smoothing are used If a schedule originally set is 
altered, for example, by a user missing a review session or moving ahead of 
the schedule set forth by the Review Module 22. 

30 Because learning, reviewing and testing will be less effective and less 

efficient if a user simply doubles up on items to be learned, reviewed or tested 
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because the user has missed a scheduled session, the system re-schedules 
Normal and Scheduled Review by re-ranking all items which still must be 
reviewed according to item importance, strength, activation, and other factors. 
This re-ordering can be done preferably using an Nth degree polynomial 
5 smoothing function. This re-ordering can also be conducted if the user, 

administrator, or system determines that the workload of any particular session 
is significantly greater or less than the sessions before or after it. It is 
desirable that the workload from session to session be as equal and uniform 
as possible to maintain the user's motivation, and to ensure the most effective 
1 0 and efficient learning, review and retrieval of knowledge and skills. 

In each of Normal Review, Ad Hoc Review and Scheduled Review of 
the Review Module 22, items are presented for review in a manner that is 
similar to the presentation of items in the Learn Module 21 to the extent that 
latency of recall is measured and calculated. Based on the measured latency 
1 5 of recall and the user's quantitative Judgement of the adequacy of his response 
to the presented cue, an item to be reviewed will either be maintained in the 
presentation group or dropped out of the review group in the Review Module 
22, The process of sorting items continues until all items are reviewed to a 
level that is desired. 
20 The time between presentation of a cue and presentation of a 

response in the Review Module 22 is preferably controlled according to user 
input, position of item within sequence of items to be reviewed, primary 
sensory modality, and other factors such as covert data taken from user, such 
as galvanic skin response, pupil diameter, blink rate etc. and other measured 
25 characteristics. The system, method and apparatus of preferred embodiments 
of the present invention also control the time between the presentation of one 
cue and one response pair and the next cue and response pair. 

In addition, the presentation of each cue and response pair in the 
Review Module 22 relative to other cue and response pairs is controlled 
30 according to timing, sequence, and fonnat of material to be presented. All of 
these factors vary over time based on user input, both overt and covert, to 
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determine which items will be presented, as well as the sequence, pattern and 
timing of presentation. 

III. THE TEST MODULE: The Test Module 23 preferably includes several 
different types of tests of varying sensitivity including a test of familiarity, a test 
of recognition, a test of recall, and a test of automaticity. Through testing and 
the use of different types of tests in the Test Module 23, the system can 
determine whether an item is known to.a user and to what degree an item is 
known (familiarity, recognition, recall, automaticity). 

In the prior art, a typical test is a test of recall in which latency of 
response is not measured and is unimportant. In contrast, in the present 
preferred embodiment, latency of response is important and is measured and 
used to modify future operations of the various modules of the system 10. 

The preferred test format is to use an alternative forced-choice test, 
preferably a five alternative forced-choice test in which a user must select one 
of the five alternatives presented in response to a presented cue. Although a 
five alternative forced-choice test is prefen-ed, it is possible to change the 
number of forced-choice responses and type of test according to various 
factors such as what level of memory strength is being measured or for what 
purpose the test is being presented. 

The Test Module 23 is important not only as a traditional measure of 
knowledge or measure of memory strength, but also because the testing in the 
Test Module 23 functions as another fomi of review. Test taking is another 
way for the user to learn, review, and to maintain motivation and interest In 
using the system 10, 

In a preferred embodiment of the Test Module 23, an item to be tested 
is presented. First the cue is presented along with a question, "Do you know 
the answer?". The user constructs a response, and then indicates his 
quantitative "feeling of knowing" by choosing one of a plurality of choices. In 
the preferred embodiment of the Test Module 23, a scale from 1 to 5 is 
presented, whereby 1 indicates that the user has no idea of the corrBct 
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response and 5 indicates that the user is absolutely certain that he knows what 
the correct response is. Scores of 2, 3, and 4 are gradated between these two 
extremes. The time period from the presentation of the scale of 1 to 5 until the 
time that the user makes his choice is measured. 

Then a plurality of forced-choice responses (preferably five) are 
presented for the user to choose from. Only one of the presented responses is 
the correct response. The time period from the presentation of the plurality of 
responses to the time when the user selects a response is measured. 

This time period is referred to as a measurement of latency of response. 
However, absolute latency is not an accurate indicator of the cognitive 
functioning of the user. Instead, relative latency is measured for each user by 
taking into account many difference latency periods, the order of presentation 
of alternative responses, the primary sensor modality of the items and other 
factors. 

After the user has selected one of the alternative choices as his 
response, the user is required to rate his response by choosing one of a 
plurality of choices in response to a question "How confident are you in your 
response?" The time between the presentation of this question and the user's 
response is measured. The incorrect responses are removed from the screen, 
leaving the correct response and the cue displayed. If the correct response 
was selected, the cue and response remain for a period of time which is 
shorter than the period of time in which the cue and response remain if the 
user chose the incorrect response. 

In addition to knowing whether a response was correct or incorrect, the 
user is provided with Information about their metacognitive judgements of 
"feeling of knowing" and "confidence of response." ThisJnformation about 
metacognitive performance is only used to assist the user in improving his 
metacognitive abilities thus improving self-paced study skills and thereby 
making the user a better learner. 

According to preferred embodiments of the present invention, items to 
be learned, reviewed or tested are presented in a sequence which is not- 
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determined by the user's metacognitive performance and perceived knowledge 
of those items, as is done in some conventional methods. That is, the 
sequence of the items in each group are presented to the user in each of the 
Learn Module 21 , the Review Module 22 and the Test Module 23 without ever 
5 querying the user as to whether the user thinl<s or perceives he knows the 
correct response or answer. Thus, the items to be learned, reviewed and 
tested are presented based on the predetermined grouping and sequencing of 
those items and the grouping and sequencing is not based on the user's 
perception as to whether the items are known or unknown. 

10 it is prefen-ed that the conditions of retrieval in the Test Module 23 most 

closely model the actual real world test or retrieval situations which the user is 
preparing for. Thus, the Test Module 23 is preferably configured to the form of 
the actual anticipated test or retrieval situations to enhance the retrieval 
practice effect. The act of retrieving an item firom memory facilitates 

1 5 subsequent retrieval access of that item. The act of retrieval does not simply 
strengthen an item's representation in memory, it also enhances the retrieval 
process. 

In terms of the presentation or sequence of items in the Test Module 23, 
it is preferred that the presentation of test items is based so as to reduce the 

20 process of elimination effect. This effect describes a method used by students 
to "learn" information early in a test that assists them in responding to items 
later in the test. In order to reduce or eliminate this effect, the most difficult 
and confusable items, for instance, are presented early in the test in the Test 
Module 23. Ordering of items is preferably based on difficulty, confusability 

25 and other suitable factors in the Test Module 23. 

The measurements of latency of response for the feeling of knowing 
judgements and the confidence in response judgements, not the actual scores 
of feeling of knowing and confidence of response, are used for scheduling 
future learn, review and test activities in the Learn Module 21 and the Review 

30 Module 22. 
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The Test Module 23 is preferably adapted to modify or normalize the 
feeling of knowing and confidence of response choices. If a user selects only 
3s, 4s and 5s, for instance, the system 10 will normalize such responses Into a 
1 , 2, 3, 4, 5 scale. The absolute judgement is important, however, and 
valuable infomiation can be obtained by measuring and calculating the relative 
values of the judgements as well. 

For the review of missed items presented in the Test Module 23, the 
sequence is determined by the relative degree of difficulty of items. Degree of 
difficulty is determined by the correctness of the user's response, the latency 
of response in providing feeling of knowing and confidence of response 
Judgements and is not based on the actual scores of feeling of knowing and 
confidence of response. Ordering the sequence of missed items on this toasis 
creates higher memory strengths of items missed In the testing. 

The system 10 can determine the user's motivation by monitoring the 
user's performance data in the Learn Module 21, the Review Module 22 and 
the Test Module 23, as well as system usage including a user's ability to 
adhere to a set schedule, how many sessions or days a user has missed, and 
other factors. 

Based on relative motivation, determined as described above, the Test 
Module 23 preferably selects an item to be tested so as to increase a user's 
motivation and confidence. The Test Module 23 is also arranged to allow for 
use of testing as a form of motivation, to break up monotony, and to use test as 
form of review. 

The date of tests in the Test Module 23, including using testing as a 
form of review, can be determined by the user, the system 10, the 
administrator, or other input sources. For example, a teacher may want to use 
a test in the Test Module 23 as a form of review when an actual classroom test 
will occur soon. 

Test as a form of review is preferably done when the strength of Items is 
relatively high and the activation is relatively low. A test as a form of review 
breaks up monotony, maintains a review schedule, allows a different form of 



wo 01/50439 

PCT/USOO/35381 



48 

retrieval practice, closely mirrors the conditions of an actual test, and may 
have a motivational influence. In addition, the scheduling of testing in the Test 
Module 23 as a form of review may be influenced by a user's performance in 
the Learn Module 21 or the Review Module 22. For example, if the user's 
5 performance in the Learn Module 21 and the Review Module 22 are less than 
desired, a test may be scheduled as an alternative form of review and also to 
Increase motivation. 

In addition, in the Test Module 23, the user, the administrator or the 
system can determine when a test should be administered. The Test Module 

1 0 23 preferably takes Into account all testing factors like time of day, gender, 
age, other personal factors including physiological measures, measures of 
attentlveness or other brain states and other environmental conditions. The 
Test Module 23 also takes into account the material to be tested, its difficulty, 
and other factors such as recency, frequency and pattern of prior exposure to 

15 material in the past. 

After testing in the Test Module 23 further review and testing may be 
scheduled based on the performance in the Test Module 23. For example,- 
items that were determined to be well known are tested and reviewed less in 
the future. Further, the system changes hopping tables for items to be 

20 reviewed and tested in the future based on latency of response, actual 
knowledge and other factors observed during the test. 

Many different fonns of tests may be used in the Test Module 23 
including a test of recall, an alternative forced-choice test, and other types of 
tests. Latency of response is preferably measured when using a test of recall 

25 or alternative forced-choice test. 

Also, Items can be tested backwards and forwards In the Test Module 
23. That is, the cue becomes the response and the response becomes the 
cue. Further, a distracter, which is an alternative forced-choice that is 
incorrect, may be used to increase testing difficulty. A distracter should be 

30 chosen from a group of similar items although not necessarily from the same 
lesson. 
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Also, confusable items are tested consecutively and may be used as 
reciprocal distracfers. The Test Module 23 determines whether users are stiH - 
confusing these items by analyzing latency of response, confidence, and by 
the user choosing the incorrect confusable response, rather than the correct 

5 response itself. Other factors may also be considered in determining whether 
items are confusable. 

IV. THE SCHEDULE MODULE: The Schedule Module 25 is preferably 
provided to interactively and flexibly schedule the operation of the Learn 
1 0 Module 21 . the Review Module 22 and the Test Module 23. The preferred 
embodiments of the present invention are set up such that a user's 
performance in the Learn Module 21 , the Review Module 22 and the Test 
Module 23 may affect operation of any of the others of the 
Learn Module 21, the Review Module 22 and the Test Module 23 to make 
15 learning, reviewing and testing more efficient and effective. 

Furthermore, the Schedule Module 25 may schedule presentation of 
items in any of the Learn Module 21 , the Review Module 22 and the Test 
Module 23 based on input information from the user, the administrator, the 
system or other input sources and other input information, including date of 
20 test or date that l<nowledge or skills are required, the current date, the start 
date, what knowledge or skills need to be learned between the start date and 
the test date, desired degree of initial learning and retention, days that study or 
learning cannot be done, how closely a person follows the schedule already 
created by the system, and many other factors. 
25 That is, the system 1 0, and the Schedule Module 25 in particular, is 

responsive to user perfonnance and user activity both within the system and in 
the real world. 

The Schedule Module 25 schedules the presentation of items in the 
Leam Module 21, the Review Module 22 and the Test Module 23 by spreading 
30 the material out to reduce cognitive workload on a micro level and a macro 
level to maximize strength and activation of all Items or skills on the 
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predetermined date. In addition, the most significant way to drastically reduce 
the cognitive workload on the user or student is to eliminate the burden of 
scheduling, determining the pattern, sequencing, and 
timing of presentation, and presenting cues and monitoring responses in the 
Learn Module 21. the Review Module 22 and the Test Module 23. which the 
Schedule Module 25 does. 

In one example of a preferred embodiment of the present invention a 
user or administrator Identifies content which is either already in the system or 
.nput thereto. The user or administrator, or system, may identify and input to 
the Schedule Module 25 the date of test, or date that knowledge or skills are 
required, the desired level of retention, the starling date, dates where no 
activity will be done, time available during each study session, whether or not a 
Final Review is desired, how well the user can perfonn according to a 
schedule, how much time is required by the user to team, review and test an 
15 Item based on past performance, and other factors. 

The system and more particularly, the Schedule Module 25 generates a 
customized schedule based on inputs from the user or administrator as noted 
above and any of the following factors: the spacing effect, strength, activation 
when a lesson was initially learned, the degree of difficulty of items the 
confusability of items or other factors upon which the Learn Module 21. the 
Review Module 22 and the Test Module 23 are based. 

The Schedule Module 25 also preferably determines whether items are 
being scheduled for presentation during a Normal Zone, a Compression Zone 
or a Final Review Zone. In the Normal Zone, an average or noma\ schedule 
of leam. review and test is conducted since there is enough time remaining 
before the test date or the date that the knowledge or skills are required to 
achieve the desired degree of strength and activation. However, during the 
Compression Zone, the Schedule Module 25 must provide more opportunities 
to review items than in the Normal Zone, That is, the Schedule Module 25 
treats items learned in the Compression Zone.as though they are more 
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difficult, increasing the number and type of reviews, so as to Increase the 
strength of those items before the Final Review. 

In addition, the Schedule Module 25 preferably uses workload 
smoothing to avoid any relative busy or easy study sessions for learning, 
reviewing and testing items. Graceful degradation also takes Into account the 
user's actual use of the system 10. For instance, if the user skips one or more 
study sessions, or gets ahead of the spheduJe. or changes the date of the test, 
or makes other modification to the input factors, the Schedule Module 25 will 
recalculate the learning, reviewing and testing that must be conducted In the 
future to ensure the most effective and efficient use of time to develop the 
desired degree of strength and activation of knowledge or skills by the 
predetermined date. 

V. THE PROGRESS MODULE: The Progress Module 26 is preferably 
provided in the main engine to quantitatively monitor performance of other 
modules, most notably, the Learn Module 21, the Review Module 22 and the 
Test Module 23. As noted above, the progress in any one of the Learn Module 
21, the Review Module 22 and the Test Module 23 may affect the scheduling 
and operations of any of the others of the Learn Module 21 , the Review 
Module 22 and the Test Module 23. 

In addition, it is important to give the user or student proper motivation 
and feedback regarding their metacognitive skills as described above, as well 
as their usage of the system. Thus, the Progress Module 26 evaluates in any 
of the Learn Module 21. the Review Module 22. the Test Module 23. and the 
Schedule Module 25 and other elements of the system such as the ' 
Discriminator Module 28. 

VI. THE DISCRIMINATOR MODULE: The Discriminator Module 28 is 
preferably provided in the main engine 20 and interacts with at least one and 
possibly each of the Learn Module 21, the Review Module 22 and the Test 
Module 23. The Discriminator Module 28 is designed to teach confCisable 
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Items. Confusable items are two or more items which are somehow similar or 
easily cxsnfused by the user, particularly in retrieval. 

Confusable items may be previously determined by the system or may 
be identified by the user, the administrator or the system during use of the 
system. 

According to a preferred method of the Discriminator Module 28, 
confusable items are an-anged in the Learn Module 21 such that a user learns 
the first and second confusable items and practices the ability to discriminate 
between the two. 



10 



If two items are confusable or difficult to discriminate., an aspect or 
feature of that item or items which increases discriminability should be 
Identified and used to practice discriminating between the confusable items. 
Preferably the user, the administrator or system identifies the aspect or feature 
that allows the confusable items to be differentiated from each other using the 
15 Discriminator Module 28. 

The Discriminator Module 28 is preferably set up to make the 
discrimination between the two confusable items as easy as possible. For 
example, visually similar items may be differentiated using a blink comparator 
which overiays and alternatively displays two items in the same position using 
20 different colors, shades, or graphical information to show clear differences 
between the two confusable items. 

It should be noted that confusable items can be a pair of items to be 
learned or an item to be learned and another item that is not scheduled to be 
learned but is conftjsable with the item to be learned. In addition, there may 
25 be more than two confusable Items which are identified and controlled by the 
Discriminator Module 28. However, it is prefen-ed that the number of 
confusable Items to be learned , reviewed and tested Is two. 

It is also preferred that the confusable pair is presented always in the 
same lesson set, review set and test set. 
30 In addition to its applications to the Learn Module 21, the Discriminator 

Module 28 also preferably interacts with the Review Module 22 and the Test 
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Module 23. For example, in the Review Module 22. confusable items may be 
reviewed together using the blink comparator. This may also be true at the 
end of a test in the Test Module 23. 

Confusable items to be learned, reviewed or tested can be presented 
using a blink comparator or other suitable ways. For example, if items are 
visually similar, the cues and responses are shown together allowing the user 
a period of time to compare the two items which are confusable. A "blink" 
button is provided to initiate the presentation. The presentation includes 
displaying the first response for a period of time, then replacing the first 
response with the second response for a period of time, 
and then repeating this process. In this way, the images seem to "blink," 
highlighting the most significant difference between the two. Further, it Is 
preferred to change the rate of presentation of overlays, order of overlays, or 
other aspects of the blink comparator. 

In addition, to further develop and retain the ability to discriminate 
between the confusable items, tests may be also be provided. When testing, a 
single cue is selected from one of the confusable items. All of the confusable 
responses are then presented. The user must choose the correct response to 
the presented cue. The correct response is then highlighted while the wrong 
responses disappear. This testing of each of the cues individually with the 
entire set of responses continues until the latency of responses and accuracy 
of responses reaches the desired criteria as shown 
in Fig. 27, which will be described in more detail below. 

It is also possible in any of the Leam Module 21. the Review Module 22 
or the Test Module 23 to change the presentation of confusable pairs by 
reversing the cue and response for each confusable pair until the user 
achieves a desired number of correct responses with a stable latency of 
response. Latency of response is preferably measured during use of the 
Discriminator Module 28 to determine relative latency and whether the actual 
relative latency is within desired limits. Also, alternative confusable pairs may 
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randomly be. dropped out of the sequence using criteria of performance and 
latency of response factors. 

It is possible to use known confusable items as known items to take 
advantage of the spacing affect to schedule presentation of unknown 
5 confusable items. The unknown confusable items can be spaced out for 
presentation of these items for better learning of differences between the two 
confusable items and to practice how to discriminate between the two 
confusable items. 

It is prefenred that confusable items are presented together in each of 
1 0 the Learn Module 21 , the Review IVIodule 22 and the Test Module 23. That is, 
It is preferred that if the user, administrator or system identifies confusable 
items, the confusable items will always be learned, reviewed and tested 
together even if the confusable items are not part of the same lesson, review 
group, or test group. Confusable items are bound together until it has been 
1 5 determined by the user, the administrator or the system that the items are no 
longer confusable. 

Now preferred embodiments of various applications and operation of 
the various modules of the system of Fig. 1 will be described. 

Fig. 7 is a flowchart showing a preferred operation of the Learn Module 
20 21 included in the system of Fig. 1. 

As seen in Fig. 7, a prefen-ed embodiment of the Learn Module is 
operated such that a sequence of items to be learned, such as the sequence 
shown in Fig. 16, is generated at step 700. The Learn Module 21 begins at 
step 700 with the generation of a sequence of items to be learned and various 
25 timing parameters of presentation of those items. The timing between 
presentation of a cue and a response is determined 

for each of a plurality of paired-associates consisting of a cue and a response. 
In addition, the timing between presentation of sets of paired-associates is 
determined at step 700. Other timing parameters such as those shown in Figs. 
30 17 and 1 8, described below, may also be determined at step 700. 
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After the sequence and timing of items to learned are generated at step 
700, the display of items to be learned begins at step 702. First, an unknown 
cue and response are displayed or presented to the user at the same time, 
step 704. Then the display is cleared of the cue and response or nothing is 
5 presented to the user, step 706. A value of N is then set equal to 1 , step 708. 
Then the cue of an unknown item Un to be learned is presented or displayed, 
step 710. The response corresponding to the cue of the unknown item Uu is 
displayed or presented to the user, step 712. Then the cue and response of 
the unknown item Un remain on the screen or are continued to be presented to 

10 the user, step 714. After this step, the screen is then cleared or nothing is 

presented to the user, step 716. A value of M is then set to 0, step 718. Then, 
a cue of a known item Km is presented to the user or displayed, step 720, 
followed by the presentation of the corresponding response of the known item 
Km, step 722. Then the cue and response for the unknown item Km remain or 

15 are continued to be presented to the user, step 724. Then the screen is 
cleared or nothing is presented to the user, step 726. 

As shown by the intermpts at steps 736 and 738, the user can interrupt 
the flow from steps 712 to 716 and from steps .722 to 726, at any time. More 
specifically, if the user interrupts the process at any time between steps 712 to 

20 716 or interrupts the process at any time between steps 722 to 726, the flow 
proceeds to step 740 at which an item is designated as having been learned 
and therefore, that item is stored as a known item in a "known" register. After 
the known item is stored, at step 740, a determination is made as to whether 
the last item has been learned, step 742. If the last item has been learned, the 

25 process flows to Quick Review, step 744, which is described in more detail 
with respect to Figs. 8A and 8B. If it is not the last item to be learned at step 
742, the user is queried as to whether they want to proceed more slowly or 
quickly, step 746, and then the process flows to step 748 where the next item 
to be learned is obtained and the flow returns to step 700 for generation of a 

30 new sequence for the next item to be learned. 
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If there is no user interrupt at steps 736 or 738, the process flows 
normally from step 726 where the display is cleared or nothing Is presented to 
the user, to step 728 where the value of M is increased by 1 . Then a 
determination is made whether M is equal to a value of N. step 730. If M is not 
5 equal to N, the flow returns to step 720 at which another known item Km is 

presented to the user. If M is equal to N, a determination is made whether N is 
equal to some predetermined number, such as, for example. 9, step 732. If N 
is not equal to the predetermined number, the value of N is increased by 1, 
step 734. and the flow returns to step 712 for presentation of another unknown 
10 item to be learned Un- If N is equal to this predetermined number, a user is 
asked whether he wants to see the next item, step 750. If a user chooses to 
see the next item to be learned, the flow returns to step 748 and 700 for 
presentation of more items to be learned. If a user chooses not to see the next 
item to be learned, the flow returns to step 702. If there is no response within 
15 a certain period of time, step 752, the process stops at step 754. 

Figs. 8A and 8B show a preferred embodiment of Quick Review which is 
part of the Learn Module 21 of the system 10 shown in Fig. 1. 

As seen in Fig. 8A, a preferred embodiment of Quick Review of the 
Learn Module 21 is operated such that a sequence of items to be Quick 
20 Reviewed is generated at step 800. The Quick Review of the Learn Module 
21 begins at step 800 with the generation of a sequence of items that have just 
been learned and are to be Quick Reviewed, and the generation of various 
timing parameters of presentation of those items. The timing between 
presentation of a cue and a response is determined for 
25 each of a plurality of paired-associates consisting of a cue and a response. In 
addition, the timing between presentation of paired-associates is determined at 
step 800. Other timing parameters different from but similar to those shown in 
Figs. 17 and 18, described below, may also be determined at step 800. 
After the sequence and timing of items to Quick Reviewed are 
30 generated at step 800. the display of items to be Quick Reviewed begins at 
step 802. First, an unreviewed cue and response are displayed or presented 
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to the user at the same time, step 804. Then the display is cleared of the cue 
and response or nothing is presented to the user, step 806. A value of N is 
then set equal to 1 . step 808. Then the cue of an unreviewed item Un to be 
learned is presented or displayed, step 810. As is seen in the parentheses of 
step 810. the cue of unreviewed item Un is actually the answer of the paired 
associate just learned. The response corresponding to the cue of the 
unreviewed item Un is displayed or presented to the user, step 81 1. As seen 
in the parentheses of step 81 1 , the response of the unreviewed item Un is 
actually the question of the paired associate just learned. After this step, the 
cue and response remain on the screen or are continued to be presented to 
the user, step 81 6.. A value of M is then set to 0. step 81 8. Then, a cue of a 
reviewed item Rm is presented to the user or displayed, step 820, followed by 
the presentation of the corresponding response of the reviewed item Rm, step 
821 . The cue and response remain on the screen or are continued to be 
presented to the user, step 822. Then the display is cleared or nothing is 
presented to the user, step 826. 

As shown by the interrupts at steps 836 and 838, the user can Intenxipt 
the flow at any time between steps 81 1 to 816, and between steps 821 to 826. 
More specifically, if the user intermpts the process at any time between steps 
81 1 to 816 or interrupts the process at any time between steps 821 to 826, the 
flow proceeds to step 840 at which a determination is made as to whether an 
item has been seen or reviewed only one or twice. If the item has only been 
reviewed one or two times, the flow proceeds to step 842, described later. If 
the item has been reviewed more than two times, the item is stored in the drop- 
out register, step 841, and then a determination is made whether the last item 
has been Quick Reviewed, step 842. If the last item has been Quick Reviewed, 
a determination is made whether four rounds of Quick Review have been 
completed, step 844. If four rounds of Quick Review have been completed, 
review curves, described later, are calculated, step 847 and the process stops 
at step 860. If four rounds of Quick Review have not been completed, a 
detemilnation is made whether there are any items which have been stored in 
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a "show again" register, step 845. If there are no items to be shown or 
reviewed again, the process flows to step 847 where review curves are 
calculated and then the process stops at step 860. If there are items to be 
shown or reviewed again, the process begins the next round of Quick Review, 
5 step 849, and the process flows to step 848 where the next item to be Quick 
Reviewed is selected. The sequence and timing of presentation for the next 
item to be Quick Reviewed is then generated, step 800. 

If there is no user interrupt at steps 836 or 838, the process flows 
normally from step 826 where the display is cleared or nothing is presented to 
1 0 the user, to step 828 where the value of M is increased by 1 . Then a 

detemiination is made if M is equal to a value of N, step 830. If M is not equal 
to N, the flow returns to step 820 at which another known item Rm is presented 
to the user. If M is equal to N, a determination is made vi/hether N.is equal to 
some predetermined number, such as, for example, 9, step 832. if N is not 
equal to the predetermined number,, the value of N is increased by 1, step 834, 
and the flow returns to step 810 for presentation of another unknown item to be 
learned Un. If N is equal to this predetermined number, a user is asked 
whether he wants to see the next item, step 850. If a user chooses to see the 
next item to be learned, the flow returns to step 848 and 800 for presentation 
20 of more items to be learned. If a user chooses not to see the next item to be 
learned, the flow returns to step 802. If there is no response within a certain 
period of time, step 852, the review curves are calculated for items in the 
"drop-out" register and other items are treated as 
if those items have been Quick Reviewed through all four rounds of Quick 
25 Review without dropping out and then the process stops at step 854. 

Fig. 9 is a flowchart for illustrating an operation of the Review IVIodule 
22 according to a preferred embodiment of the present invention. As seen in 
Fig. 9, the Review Module 22 begins by displaying a cue and asking a user 
whether he wants to see the answer yet, while also beginning a timer, as 
30. shown in step 900. During this time, the user is expected to construct or 
formulate a con-ect response to the cue presented in step 900. The user is 
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expected to construct or formulate the correct response within a certain period 
of time, STAn. If a user interrupts the operation of the Review Module before 
the period of time STAn lapses, step 902, the cue is displayed with a paired 
response, step 904. Then the screen is made blank and a response to a query 
5 asking the user to quantitatively rate the quality of his response is requested 
while the timer is started, step 905. The user is expected to rate the quality of 
his response within a certain time period RTQ^. If a user does not intenxipt 
operation before the time period RTQn has lapsed by providing the rating of 
quality of response, step 908, the screen is made blank, step 910, and that 
1 0 particular item is transferred to a storage register Sn^i and flow proceeds to 
step 920. Sn or Sn+i represents a storage register where items for which the 
user either could not identify the correct response or had trouble in identifying 
the correct response as indicated by a low rating of the quality of his response, 
are stored for additional review in the future. The variable n in Sn or Snti 
1 5 indicates the number of the pass or round of Review. If the user does Interrupt 
the operation of the Review Module 22 after step 905, before the period of time 
RTQn has lapsed, by providing the response to the request for rating his 
response, step 912, a determination is then made whether the user has rated 
his response to be high quality (e.g. a value of 4 or 5) or low quality (e.g. a 
20 value of 1 , 2 or 3). If a low quality response is provided, the control proceeds 
to step 914 described above so that the item receiving a low quality rating is 
stored for future review in the register Sn*i. If the user rates his response as 
high quality, the control proceeds to transfer to at step 916 and then the 
screen is made blank at step 91 8 and flow proceeds to step 920. □„ 
25 represents a discard register where items which are well known to the user, as 
indicated by the high quality response, are stored and are not reviewed again 
in another round of the Review Module 22. At step 920, the determination is 
made whether Sn, the storage register with the items receiving low quality 
performance ratings, is empty. If Sn is not empty, meaning there are more 
30 items to be reviewed, the presentation may be paused at the user's request, 
step 922, and then control returns to step 900 for further operation. If Sn is 
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empty meaning there are no more items to be reviewed, it is determined at 
step 924 whether N is 4. N is a value indicative of the number of rounds of 
Review, or can be thought of as the number of times a user has reviewed all of 
the Items in the storage register Sn. if N is not 4. N is increased by one at step 

5 926 and the flow returns to step 922 to return to the beginning at step 900 after 
a brief pause at step 922. If N Is equal to 4 meaning the user has made four 
passes through Review, the user is asked if he wants to relearn ail items that 
remain in the S„ register, step 928. If a user chooses to relearn a particular 
item, the flow Is transferred to the Learn Module 21 at step 930. If a user 

0 chooses not to relearn an item, the control exits out of the Review Module 22 
at step 932. 

If the user falls to interrupt the Review Module 22 before the time period 
STA„ has lapsed by failing to request that the answer or response be shown, 
step 950, the cue Is displayed with the paired response at step 952. Then the 
screen Is made blank, the cue is again displayed by itself, a response is 
requested and a timer is started, step 954, which Is similar to step 900. If the 
user does not interrupt before the time period STA, lapses, that is the user did 
not request that the answer be shown, at step 956. the flow returns to step 952 
In which the response is shown with the cue. If the user does interrupt before 
the time period STAn lapses, step 958, the cue is displayed with the paired 
response, step 960. and the flow proceeds to step 962 at which point the 
screen Is made blank, a response for rating the quality of response is 
requested and the timer for timing the time period RTQ„ is started. If a user 
interrupts before the time period RTQ^ lapses, that is before the user rates the 
quality of his response, step 964, the response Is ignored and the screen Is 
wiped blank at step 968. If the user does not Interrupt before the question is 
repeated at step 966. the response is ignored and the flow proceeds to step 
968. The response is Ignored In both cases because it has already been 
determined that this particular item should be reviewed again. Then the item is 
placed in the register Sn*i at step 970 and flow proceeds to step 920. and 
further processing occurs as described above. 



wo 01/50439 

PCT/USOO/35381 



61 

Figs. 1 0A and 1 B are flowcharts for illustrating an operation of the Test 
Module 23 according to a preferred embodiment of the present invention. As 
seen in Fig. 10A, the Test Module 23 begins by the user selecting an Ad Hoc 
Test, step 1000 or by the system 10 displaying a test button in the main menu 
5 display, step 1 002, for a Scheduled Test. Then the user selects or taps on the 
test button on the display, step 1004, and the items for testing are selected 
and a sequence of items to be tested is generated, step 1006. The first cue of 
the items to be tested is then presented and a timer is started, step 1 008. In 
this prefen-ed embodiment, the user is asked to select a "feeling of knowing" 
1 0 score, for example, by indicating on a scale of 1 to 5 how confident the user is 
that he knows the correct response to the cue. The user selects the feeling of 
knowing score and the timer. Is stopped at step 1010. Then the cue is 
displayed with preferably 5 alternative forced-choices and a second timer is 
started, step 1012. The user then selects one of the 5 alternative forced- 
1 5 choices and the second timer is stopped, step 1014. If the response selected 
by the user is correct, the incorrect answers are eliminated from the display 
and an audible signal is produced and then the con-ect response is highlighted 
and shown for a time T3, step 1 018. If the response selected by the user is 
not correct, the incon-ect answers are eliminated from the display and an 
20 audible signal is produced and then the correct response is highlighted and 
shown for a time T4, which is longer than time T3, step 1016. Then the correct 
answer position and the selected answer position are saved as are the feeling 
of knowing score and the accuracy of response, step 1020. Then it is 
detemiined whether the item just tested was the last in the sequence of items 
25 to be tested, step 1 022. If there are more items to be tested, the user is 
allowed to pause and then the operation returns to step 1008 for testing of 
more items, step 1 024. If there are no more items to be tested, the test scores 
are calculated and displayed, and a user is asked if he wants to releam the 
items that for which the user selected the incorrect response, step 1 026. If a 
30 user chooses to releam missed Items, the missed Items are releamed using 
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the Learn Module 21, step 1028. as described above. If the user chooses to 
not relearn missed items, the Test iVIoduie stops, step 1030. 

Fig. 1 1 is a flowchart of an operation of a preferred embodiment of the 
Schedule Module 25 preferably provided in the system of Fig. 1 . The 
Schedule Module 25 begins at step 1100 at which information relating to the 
Schedule Module 25 is input or updated. The information to be input at step 
1100 may preferably include the start date, the end date, the lessons to be 
learned, reviewed and tested, the types of lessons, the desired level of 
retention, the amount of time each day that the user is available to use the 
system, the number of final reviews, the time available for final reviews, the 
user's history of system usage, black out days when use of the system is not 
possible, and other factors and Infonnation. After this Information is input at 
step 1 100, the final review zone is calculated at step 1 105 so as to determine 
the start date and end date of the final review period. Then the compression 
zone is calculated at step 1 1 1 0 to determine when the compression period 
begins and ends. After this, the nonnal zone is calculated at step 1 115 to 
detennine the start and end dates of the normal period. Then the system 10 
checks for the presence of scheduling errors at Step 1 120. Scheduling errors 
include the scheduling of too many items within too short of a time period 
based upon the demonstrated ability of the user or other input. Other en-ors 
may also be checked for. If scheduling errors are detected at step 1 120, a 
warning is issued to the user at step 1 122. If a user chooses to modify the 
input information to avoid such scheduling errors at step 1 124, the flow returns 
to step 1100 to begin the Schedule Module 25 again and re-calculate the 
schedule. If a user chooses to proceed with the Schedule Module 25 despite 
the presence of scheduling errors at Step 1 120, the flow proceeds to generate 
a schedule at step 1 128. As described above, the schedule is generated 
based on the input information including the user's past history and usage of 
the system 1 0 and ability to comply with previously generated schedules. After 
a schedule is generated at step 1 128, the schedule is checked for workload 
smoothing at step 1 130 to avoid any session or day in which too much work or 
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not enough work is scheduled relative to the preceding or following days. The 
schedule may be modified at step 1 130 to achieve sufficient workload 
smoothing. Then, the user's progress with the system and specifically, the 
user's ability to comply with the generated schedule is monitored and stored in 
the system at step 1 132. The system detects at step 1 134 whether there is 
any deviation firom the schedule generated at step 1 128. If there is any 
deviation from the schedule, the control returns to step 1 100 for re-calcuIation 
of the schedule to accommodate and compensate for such deviations. If there 
is no deviation from the schedule, the flow proceeds to step 1 136 in order to 
detemiine if the final review start date has arrived. If the final review start date 
has not arrived, the flow returns to step 1 130 to further monitor progress and to 
detect any deviations from the schedule. If the final review start date has 
arrived, the Schedule Module 25 generates a final review schedule based on 
relative difficulty of the items, the recency, the frequency, the pattern of prior 
exposure and other factors, step 1138. The user's performance in final review 
is monitored and controlled at step 1 140 until the end date at which time the 
Schedule Module 25 ends, step 1 142. 

Figs. 12, 13A and 13B each show a flowchart of an operation of a 
preferred embodiment of the Discriminator Module 28 preferably provided in 
the system 10 of Fig. 1 . The Discriminator Module 28 begins with either a 
Scheduled Discrimination review or test, step 1200, or with an Ad Hoc 
Discrimination review or test, step 1202. The process then begins at step 
1204 and confusable items are displayed or presented to a user in 
a side by side or closely associated presentation, step 1206. The user then 
decides whether to compare the confusable item or to be tested on their 
knowledge of the confusable items, step 1208. If the user chooses to 
compare the confusable items, the responses of the confusable items are 
displayed or otherwise presented to the user to allow the user to compare and 
discriminate differences between the confusable items, step 1212. If a user 
interrupts this process, step 1214, the user is provided the choice 
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Of being tested, moving to the next item or quitting operation of the 
Discriminator Module 28. If a user chooses a test at step 121.4, the flow- 
proceeds to step 121 0. If the user chooses to end operation of the 
Discriminator Module 28 at step 1214, the operation of the Discriminator 
5 Module 28 stops at step 1216; If a user chooses to move to the next 

confusable item, the flow returns to steps 1200. 1204 and the next group of 
conf usable items is presented at step 1206. 

At step 1208. if the user chooses to be tested on the confusable items, 
the flow proceeds to step 1210 and the process shown in Figs. 13A and 13B. 

0 As shown in Fig. 13A, if a user chooses to be tested on confusable 
items, test fonns and sequences are generated at step 1300. Then various 
test forms are selected from the total set of test forms for use in presentation to 
the user, step 1 302. Then a cue is presented to the user with various 
response choices and a first timer is started, step 1 304. Then a user selects 

1 the response he believes to be the correct one and the first timer is stopped, 
step 1306. If the response is correct, the incorrect responses are removed 
from the display and the cue and correct response remain displayed for a 
certain, period of time X with the correct response being highlighted and an 
audible signal is presented, step 1310. If the response is not correct, the 
incorrect responses are removed from the display and the cue and correct 
response remain displayed for a certain period of time Y, longer than the 
period of time X, with the correct response being highlighted and an audible 
signal is presented, step 1308. Then the test form is erased or removed from 
the display, step 1312. A determination is then made if the last test form has 
been presented to the user, step 1 31 4. If there are more test fonns to be 
presented to the user, the control returns to step 1302 for presentation of more 
test forms for testing confusable items. If there are no more test forms to be 
presented to the user, a determination is made whether all of the test forms 
were answered correctly, step 1316. If not all of test forms were answered 
correctly, a determination is made whether it is the fourth set generated for the 
particular items being tested, or the fourth time that those particular confusable 
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items were tested, step 1318. If it is not the fourth set or fourth time, the 
control returns to step 1 300 for generation of another set of test forms. If it is 
the fourth set or fourth time, the process stops at step 1320. If all of the test 
forms were answered correctly as determined at step 1316, a determination is 
made whether it is the first set or first time that the set of test forms was 
generated for this particular group of confusable items, step 1322. If it is the 
first set or first time, the control returns to step 1300 for generation of another 
set of test forms. If it is not the first set or first time, a determination is made as 
to whether the average time for response for the current set of test forms is 
shorter or less than previous time for response, step 1324, and if so, the 
process ends at step 1326. If the average time for response for the current set 
of test forms is greater than or equal to the previous time for response, the flow 
returns to step 1300 for generation of another set of test forms. 

The sequence of items to be learned in the Learn Module 21 
generated in step 700 of Fig. 7 may be generated based on the input desired 
degree of initial learning or level of learning. Fig. 14 shows various levels of 
learning possible in the system 10 of preferred embodiments of the present 
invention, along with a graph of memory strength versus time which includes 
the forgetting/retention curve shown in Fig. 3. As seen in Fig. 14, four levels of 
learning are located at various points along the forgetting/retention curve 
shown in Fig. 3. In the order of lowest learning level to highest learning level, 
the four levels of learning are: familiarity, recognition, recall and automaticlty. 

Information learned or remembered to the level of familiarity is 
information that the user has the feeling that they knew at one time, but can no 
longer remember. 

Information learned or remembered to the level of recognition is 
information that the user can separate from other distracting choices or 
distracters. When presented with a cue, the user can choose the appropriate 
response from a number of alternatives. For example, the user may be asked 
select the correct answer on a multiple choice test 
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Information learned or remembered to the level of recall is Inforrriation 
that the user can retrieve when only a cue is presented. For example, the user 
may be asked to provide the correct response to a provided cue on a "fill In the 
blank" test. 

Information learned or known to the level of automaticity is infonnation 
that the user can retrieve instantly, with little or no cognitive effort, when only a 
cue is presented. The user "knows" the information as opposed to 
"remembers" the information. Automaticity can be measured by a test of recall 
where accuracy is required and latency of response is the key variable. 

As shown in Fig. 3 and described above, previously learned items such 
as knowledge or skills, are gradually forgotten over time. The higher the level 
of initial learning, the longer the information is available for retrieval. Learned 
information passes down through the various levels until it is only familiar. 

Different types of tests have varying degrees of sensitivity. A student 
could answer a question correctly on a multiple choice test, but miss the same 
question on a test of recall. Therefore, a test of recognition is less sensitive 
test of memory strength than a test of recall. Similarly, a test of recall is a less 
sensitive measure of memory strength than a test of automaticity. 

In preferred embodiments of the present invention, the items to be 
learned are presented in the sequence generated in step 700 of Fig. 7 in such 
a way that the user learns to a level of automaticity. The benefits and 
processes for learning to a level of automaticity will be described below. 

In order to have a user learn to a level of automaticity as shown in Fig. 
14, the system 10 presents items to be learned by taking advantage of the 
principle of overleaming as shown ih Fig. 1 5. More specifically. Fig. 1 5 shows 
the benefits of overleaming. The degree of initial learning affects future 
perfomiance as described above. The decay rate for memory is approximately 
parallel for various degrees of initial learning as shown by the parallel curves 
in Fig. 15. Material leanned to a level of mastery (100% correct on a test of 
recall) is forgotten at the same rate as overleamed material (100% correct on a 
test of recall, with low latency of response and low cognitive effort). Since both 
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curves in Fig. 15 are substantially parallel, however, at any point in the future, 
retrieval performance is. higher for overleamed material. Additionally, material 
which is initially overlearned to a level of automaticity is more likely to survive 
the initial, fragile period of consolidation where most memories are lost due to 
decay and interference. 

The parallel nature of the curves in Fig. 15 is independent of the time 
schedule. Material learned to a higher degree of initial learning has a higher 
memory strength than material learned to a lower degree of initial learning 
even when measured decades later. 

For generating the sequence of items to be learned at step 700 of Fig. 
7. the system 10 preferably determines a sequence of items to be learned and 
a time period between a presentation of a cue and a response of a paired- 
associate and a time period between presentation of successive paired- 
associates to achieve learning to a level of automaticity shown in Fig. 14 by 
using overiearning shown in Fig. 15. More specifically, the items to be learned 
in the system 1 0 are preferably arranged according to a learn presentation 
sequence shown in Fig. 16. As seen in Fig. 16. Items (cues and responses) 
presented in the Learn Module 21 of the system 10 are sequenced according 
to whether they are items to be learned or are items that have already been 
learned. In Fig. 16, items being presented for the first time are designated as 
U, where I indicates that it is the initial presentation of the unknown item. That 
same item seen again and again is designated as Un, where N is the number 
of times that the item has been previously seen within the sequence. Items 
which have already been presented during previous Learning Module 21 
operation are considered to be "known" for the purposes of sequencing and 
are designated as Kr, where the R Indicates that they are items chosen 
randomly from the pool of known items. 

By creating a sequence of known and unknown items as shown in Fig. 
16, a form of expanded rehearsal for the unknown item is created. As 
mentioned previously, the expanded rehearsal series is the most effective and 
efficient schedule of review to build memory strength. In experimental 



wo 01/50439 



PCT/DSOO/35381 



68 • 

psychology, this is known as the spacing effect. The sequence shown in Fig. 
16 creates an intra-triai spacing effect. The schedule of review described in 
Fig. 4 creates an inter-trial spacing effect. 

At some point during the presentation of the sequence shown in Fig. 16, 
5 the user will determine that they have learned the previously unknown material 
by comparing the adequacy of their response to the cue with the correct 
response provided by the system 10. When the user judges that their response 
Is adequate, the user intenupts the presentation sequence. This interrupt will 
take the user to the next unknown item to be learned if any remain in the 
0 lesson, or to a Quick Review session if all items within the lesson have been 
seen. 

Fig. 17 illustrates a learn presentation pattern for the presentation of the 
items described in Fig, 16 which is broken down and described in further detail 
in Fig. 17. 

5 At the beginning of the sequence, the unknown Item to be learned within 

the sequence is presented as U,. Both the cue and the response are • 
presented at the same time (Ti). 

The cue and response disappear leaving a blank screen, or depending 
upon the modality of presentation of the information, a null event— one where 

0 nothing happens (Ta). 

Now the unknown item is displayed or presented by itself. At this time, 
the user absorbs the cue and attempts to actively recall the appropriate 
response (T3). 

Whether the user is successful in retrieving the response or not, the 
correct response is presented using a method that may be the same as the 
method of presenting the known response, but preferably is unique to the 
presentation of the response to be learned. The method of presentation could 
involve color, sound, motion or any other method that differentiates it from the 
presentation of the randomly-chosen known response. The time that It takes to 
present the response using the defined method is called T4. 
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Both the cue and response continue to be presented. The user uses this 
time (Ts) and the time available in T4to compare the response that the user 
retrieved to the correct response. If their response is judged adequate, the 
user can interrupt the sequence and move on to the next unknown item to be 
5 learned. 

Following Tg, both the cue and response are eliminated leaving a blank 
screen or null event (Tg). 

Now a known item is selected from the group of previously learned 
items. It is presented by first displaying the cue for a short period of time (T7) 
1 0 allowing the user to attempt to actively'recall the correct response, then the 
correct response is shown according to a method that may be the same as the 
method of showing the unknown response, but preferably is unique to the 
presentation of known responses (Ta). Both the known cue and known 
response remain presented for a period of time (Tg), and then both are 
1 5 eliminated for another period of time or a null event (T10). 

The presentation pattern of showing unknown cues and responses and 
known cues and responses separated by null events preferably follows the 
sequence described with respect to Fig. 17 until the user inten-upts the 
sequence at allowable times as described in Fig. 7 or some other event occurs 
20 such as predefined time or number of presentations is reached. 

Fig. 18 shows a table indicating the presentation timing preferably used 
in the Learn Module 21 . There are preferably ten separate timing variables 
used as shown in Fig. 18, which preferably vary according to the position of 
the unknown or known cue or response within the sequence of items shown in 
25 Fig. 16. The timing parameters are set at an initial value and then are 

changed according to overt and covert responses input to or sensed by the 
system 10. One overt response to the system 10 occurs when the user 
inten-upts the presentation sequence because he wishes to learn a new item. 
At this point the question Is asked, "Do you wish to go faster or slower?" in 
30 order to maintain the attention and arousal of the user. If the user responds by 
choosing "faster* the timing values are decreased by the amount defined within 
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the table for that timing parameter. If the user responds by choosing "slower", 
the timing values are increased by the amount defined within the table for that 
timing parameter. 

The purpose of varying the timing values is to maintain the user's 
attention and arousal. Timing sequences where there is little or no variation in 
the stimulation can become habituating. That is. the stimulus is no longer novel 
and the brain tunes it out 

Additionally, each user has a desired rate of learning detennined by the 
rate of presentation of each item as well as the rate at which new items are 
presented. In the classroom, when the teacher is lecturing, all students are 
presented information at the same rate. Some students find this boring 
because the presentation is too slow, and others find it frustrating because the 
presentation is too fast— they are left behind. 

in the system 10 described above, the pattern, sequence, and timing of 
items are varied to maintain the user's interest, and provide each individual 
user with a rate of learning that each user finds challenging. Thus, the system 
10 adapts to each user. 

Also related to the sequencing of the items to be learned is a 
phenomenon knows as the serial position effect. Fig. 19 illustrates the serial 
position effect which Is a well understood phenomenon of psychology and 
involves the learning of items presented in a list. Fig. .19 shows that when 
items are presented in a list, the probability of successful recall varies based 
on the item's position within the list. If the recall test is administered 
immediately prior to the learning session, a recency effect is shown. That is. 
items presented later In the list are more likely to be recalled than previously 
presented items because the later presented items are still in the user's short 
term memory. If the recall test Is administered after a delay of several minutes, 
the recency effect disappears because the items cannot be maintained in short 
term memory for that period of time without rehearsal. This effect contributes 
to judgement of learning en-ors which detrimentally affect self-paced learning. 
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Items appearing early in tlie list are more likely to be recalled because 
of the primacy effect. Items appearing early in the list are more likely to be 
rehearsed a greater number of times than items later in the list as shown in 
Fig. 20. The success in recalling items from the list is dependent upon the 
6 number of times the item was rehearsed. 

More specifically, Fig. 20 shows a graph of the number of Rehearsals 
vs. Input Position of an item to be learned. Fig. 20 illustrates the veracity of 
the statements made in the description of Fig. 19 regarding the primacy effect. 
Items presented early in a list are rehearsed more times than items presented 
1 0 later in the list and are therefore more likely to be recalled at the time of the 
test 

Figure 21 shows a graph of memory comparison time versus memory 
span. Fig. 21 indicates that the memory span for information varies by the 
type of infonnation. Memory span for digits is approximately seven plus or 

1 5 minus two digits. That is, most people can keep seven plus or minus two digits 
in their short term memory through the process of maintenance rehearsal— 
they repeat them over and over. The rate at which a person can repeat a 
particular type of information directly affects their span for that type of 
information. This rehearsal rate varies from person to person. Generally 

20 speaking, adults can maintain more items in short term memory than children 
because their rehearsal rates are faster. Also, the language that a verbal item 
is rehearsed in affects the memory span. For example, when rehearsing digits, 
speakers of Chinese can maintain more items in memory than speakers of 
Welsh. Likewise, memory span for images, sounds, and graphical Information 

25 will vary from person to person. This phenomenon and those shown in Figs. 
19-21 are taken into account within the system 10 by the use of the modality 
pairing matrix shown in Fig. 22 which is used to define parameters associated 
with the sequence and pattern, and in particular, the timing of presentation. 

30 Fig. 22 shows a modality pairing matrix in vyhich the response follows 

the cue preferably by about 260 milliseconds to about 750 milliseconds is a 
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general guide for maximum conditioning. Some infomiation takes more time to 
be absorbed than others. The differences in time for encoding and storage of 
information are a result of the input channel or the primary sensory modality, 
the complexity of the material, the familiarity of the material, distractions to the 
use of the system by outside conditions, and many 
other factors. Fig. 22 describes the flexibility of the system 1 0 in handling 
materials presented in any combination of sensory modalities and information 
formats in both the cue and the response. The system 10 has predefined 
parameters for the presentation pattern, rate, and sequence for each 
combination of cue and response described In Fig. 22. These parameters may 
be modified by the user, administrator, or system 1 0 in 
order to create maximum conditioning adaptive to each user for each item 
learned. 

Now prefenred embodiments of the Review Module shall be discussed. 
15 As described with respect to Figs. 3 and 4, the Review Module operates 

based on the forgetting/retention function and spaced rehearsal series shown 
in Fig. 4. 

Fig. 23 shows a review curve table preferably used in the prefen-ed 
embodiments of the Review Module 22. As mentioned in the description of 

20 Figure 3, no single curve can model the forgetting rate of each item learned by 
each user. In the current preferred embodiment of the system 10, a "family" of 
curves are preferably modeled to encompass the range of forgetting: from very 
easy items to very difficult items. The curves shown in Fig. 25 have been 
sampled to create a table of numeric values. In this example, eight curves have 

25 been modeled to represent the total range. The values within the matrix shown 
in Fig. 23 indicate when a session of the Review Module 22 should occur and 
are representative of the number of days since an item was initially learned. 
Those with ordinary skill in the art can create any number of ways to represent 
the range of forgettingVetention and use the system to calculate the next 

30 session of the Review Module, based on Input from the user, to maintain any 
desired level of retention. 
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Fig. 24 illustrates a review hopping table. As noted above, since no 
single curve can accurately model the rate of forgetting, a family of curves is 
used by the system 10 to characterize the range of forgetting. Many variables 
can change overtime however, which affects the rate of forgetting, A curve 
which accurately models the forgetting rate of a particular item for a particular 
learner early in the Review schedule may become inaccurate at some later 
date due to such effects as proactive or retroactive interference and other 
factors. In order to accurately model the rate of forgetting, the system "hops" 
the item to be reviewed from one curve to another to more accurately model 
the forgetting rate. Fig. 24 shows the hopping rules that determine when an 
Item should hop from one forgetting curve to another forgetting curve shown in 
Fig. 25. During each session of the Review Module 22, the user is presented 
with items previously learned. A cue is presented and the user attempts to 
actively recall the appropriate response. After the user has made his best 
attempts, the user taps the "Show the Answer" button which causes the correct 
response to be displayed. The user is asked to rate the quality of his 
response. This rating is called the "score". 

As shown in Fig. 24, scores range from a low of 1 to a high of 5. If a 
score of 4 is given in the first round, the items changes "0" curves and is 
dropped from the current Review set. If a score of 5 is given, the item changes 
curves and is dropped from the Review set. Changing "-1" means that the 
item is moved to 1 curve "easier" than the current curve. An easier curve is 
one where Review sessions occur less frequently. Relating this to Fig. 25, the 
item may be moved from curve 4 to curve 3 — a change of 

If in the first round of the Review Module 22, the quality of response was 
scored as a 1 , 2, or 3, the item simply moves to the next round of the Review 
Module. No changes are made to the curve at this point. 

The review of each item and scoring and scoring of quality of response 
occurs round after round until no items are remaining, or until the fourth round 
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of the Review Module 22 is complete. If an item has been seen in four rounds 
and a quality rating is consistently given as 1 , 2, or 3. the item is treated as 
"unlearned", and the whole process of Learning. Review, and Testing begins 
all over again. 

This example of determining the appropriate curve to model the rate of 
forgetting of an item overtime based on scoring the quality response during a 
session of the Review Module 22 represents only one way to monitor and 
control the ever-changing rate of forgetting. The current system 10 also takes 
into account latency of response, scores on scheduled and ad hoc tests, the 
rate of initial learning, the degree of initial learning, and many other factors. 
Those with ordinary skill in the art can also create other systems based on the 
present invention that modify the model for the rate of forgetting of each Item 
for each user based on overt and covert feedback taken based 
on performance in the Learning Module 21 , the Review Module 22, and the 
16 Test Module 23 as well as data available from other sources such as the rate 
of forgetting of other users of the system or other factors. 

Fig. 25 illustrates a family of review curves with hopping. Fig. 25 
graphically represents one family of Review curves with a trace of an item 
hopping between ounces. Many different families of curves can be used by the 
system 10. Each family of curves is designed to accurately model forgetting 
for a particular type of information, knowledge or skill learned, retained, and 
retrieved. The family of curves which best model verbal infomiation may be 
different than the family of curves for auditory information. These variations in 
curves may vary from user to user as well. A family of curves which best 
model auditory information for one user may be ideal for modeling visual 
information for another user. The system 10 constantly monitor's the users 
rate of forgetting and rate of timing of "hopping" to minimize the need for 
hopping. Families of curves which result .in less hopping are considered to be 
better curves than curve families that result in more hopping. 

Fig. 26 illustrates forms for the discrimination of two items preferably 
used in the Discriminator Module 28. Fig. 26 represents the eight separate 
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forms of presenting cues and responses for two confusable items. In the first 
form, when the cue is presented as Question 1, the user should choose 
Answer 1 on the left as the correct response. Presenting the cues and 
responses for the two confusable items in the various formats, the user is 
trained to discriminate between the items in any possible scenario. Also by 
presenting the cues and responses in varying formats, the user does not get 
bored during the training session because of repetition. 

Fig. 27 illustrates the latency of response in discrimination trials 
according to a preferred embodiment of the Discriminator Module 28. Learning 
to discnminate between two items is a skill. Skills can be Improved through 
practice. One measure of performance of a skill is latency of response With 
practice, scores for latency of response decrease along a negatively 
accelerated curve, called -theoretical scores" in Fig. 27. At first, the user has a 
difficult time discriminating between the tvvo confusable Items. The user 
requires a relatively long period of time to perform this function. This time is 
known as the Upper Bound - it is the slowest the user will ever be performing 
at this Skill. With practice, the user becomes faster at discriminating between 
the Items. There Is a Lower Bound to how quickly the user can perform this 
skill based upon the limitations of perception, cognition, and reporting reflexes 
With practice, the latency of response decreases from the Upper Bound 
asymptotically to the Lower Bound. Because of the laws of diminishing 
returns, it is not desirable to continue training for too long. The decreasing 
benefit of the training does not justify the time expended. Therefore a criteria 
level IS set. When the user reaches this criteria between the Upper Bound and 
the Lower Bound, the training session is complete. Criteria levels can be set by 
the user, the system 10. the administrator or other input sources. 

Fig. 28 illustrates various schedule zones and workload for a prefen-ed 
embodiment of a Schedule Module 25 Of the present invention. Fig 28 
.ilustrates the work zones created by the Schedule Module 25 for the system 
10. The user or the administrator defines the start date, the end date and the 
Items desired to be learned. The system 1 0 automatically determines the most 
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effective and efficient scliedule of operation of the Learn Module 21, the 
Review Module 22 and the Test Module 23 to build the greatest strength and 
activation for all of the items in the curriculum by the defined end date. 

The white areas in Fig. 28 represent the number of items to be learned 
each day. The cross-hatched areas in Fig. 28 indicate the number of items to 
be reviewed each day. The black areas indicate the number of items for Final 
Review each day. 

In the Normal Zone, items are learned and reviewed in the normal 
manner. In the Compression Zone, items are learned in the normal manner 
but are reviewed as those items are particularly difficult This creates more 
opportunities to build strength of the items when very little time remains prior to 
the end date. In the Final Review Zone, all items have been learned and 
reviewed to develop the maximum strength possible. One or more Final 
Review sessions are scheduled to maximize and equalize to the greatest 
1 5 possible extent activation for each item. This presents to the user all of the 
material just prior to the end date in one oi- more of the last reviews. 

According to one prefen-ed embodiment of the present invention, a 
system 10, as shown in Fig. 1, is embodied in a processor-based apparatus 
and method in which information including items to be learned, reviewed and 
tested is presented to a user graphically, auditorily, kinestheticaily, or in some 
other manner. More specifically, the prefen-ed embodiment shown in Figs. 29- 
51 is a processor-based system 10 including a display for showing various 
window displays described with reference to Figs. 29-51. 

Fig. 29 shows one prefen-ed embodiment of the present invention, in 
26 which a Main Window display is provided to allow the user to choose which 
function he wishes to perform. As seen in Fig. 29. such functions can include 
viewing lessons, including items to be learned, within a Directory and to 
organize lessons in any way that the user desires by using any of the Find, 
New, Move or Delete options. Also, as seen in Fig. 29, the. user can select any 
one of the Learn Module 21 , the Review Module 22, the Test Module 23, the 
Schedule Module 25. the Create Module 200. the Connect Module 300. the 
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Progress Module 26 and the Help Module 28. The operation of these various 
Modules will be described in more detail below. It should be noted that other 
types of Modules may also be included in the system and the display shown in 
Fig. 29. 

5 In this preferred embodiment of the present invention shown in Figs. 

29-51, the display is preferably a touch-screen type display which responds to 
contact by a pen, stylus, finger or other object. Other types of displays or 
infonnation presentation apparatuses may also be used in various preferred 
embodiments of the present invention. 
1 0 Fig. 30 shows a Preview Window display which is presented in 

response to a user selecting a Lesson such as Lesson 1 . In one prefen-ed 
embodiment of the present invention, if a user taps twice on the display at the 
location of the title "Lesson 1" in rapid succession, the display presents 
infonnation about that lesson including the lesson's title, the author of the 
1 5 lesson, the date of creation of the lesson, and description/instructions for 
learning that lesson. In addition, the user can tap the Preview button to see 
the contents of the lesson. Tapping the Close button tal<es the user back to 
the Main Window display shown in Fig. 29. 

Fig. 31 is a display showing operation of the Learn Module 21 including 
20 the presentation of a cue. In this preferred embodiment of the present 

invention, when a lesson is selected as described above and the Learn button 
shown in Fig. 29 is tapped, the Learn Module 21 is initiated. Fig. 31 shows the 
display corresponding to T3 in Fig. 1 7. 

Fig. 32 is a display showing a further operation of the Learn Module 21 
25 including the presentation of a response to the cue shown in Fig. 31 
corresponding to T4 in Fig. 17. 

Fig. 33 is a display showing a further operation of the Learn Module 21 
including e presentation of a prompt asking the user whether he wants to 
proceed Faster or Slower. Fig. 33 shows thb window displayed after the user 
30 has determined that he knows the unknown item being presented and has 
interrupted the sequence of presentation of this particular unknown item. 
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Fig. 34 shows the display that is provided after the user has completed 
the entire process of learning a lesson. 

Fig. 35 shows another operation of the Learn Module 21 according to a 
preferred embodiment of the present invention in which a user is asked if he 
wants to learn a new item. Fig. 35 shows the window displayed when a user 
has reached a point in the presentation sequence when no user interrupt is 
given, but a predetermined time or presentation value has been reached. The 
user determines whether he wants to leam a new item or continue learning the 
item that is currently being presented for learning. If the user chooses "Yes," 
the next unknown item is presented. If the user chooses "No," the presentation 
sequence for the item currently being learned is started over again. 

Fig. 36 shows a further operation of the Learn Module 21 including the 
operation of the Quick Review part of the Learn Module 21 . Fig. 36 shows the 
display presented at the end of each Quick Review round. 

Fig. 37 shows a display including a Main Window with Review 
Notification included therein. In one preferred embodiment of the present 
invention, when items previously learned are scheduled for review via the 
Review Module 22 on the day that the user turns on the device, the Review 
button on the display is green and blinks to capture the user^s attention. Also 
shown in Fig. 37, the green icon is arranged to move and preferably spiral next 
to the lesson icon on the display indicating that the lesson has been learned 
and that such lesson has been put on a schedule of review. 

Fig. 38 shows a display illustrating operation of the Review Module 22. 
According to a prefen-ed embodiment of the present invention, when items 
previously learned are scheduled for Review on the day that the device is 
turned on, the Review button on the display is green and blinks to capture the 
user's attention. If the user has selected a lesson in the Directory, and then 
taps the Review button, the window shown in Fig. 38 appears and asks the 
user what they would like to Review, for example, items scheduled for review 
today or the lesson selected in the Directory Window. The default is the 
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Scheduled Review. The user selects one of the two and taps Continue to 
review his choice or taps Cancel to return to the Main Window. 

Fig. 39 shows a further operation of the Review Module 22. According 
to a preferred embodiment of the present invention, after the user has 
5 completed a round of Review, and the round is not Round 4, the display shown 
in Fig. 39 is presented. 

Fig. 40 shows another operation of the Review Module 22 including 
presentation of a cue. According to a preferred embodiment of the present 
invention, after the user has selected a lesson to Review or has selected to 
10 review items scheduled for Review, he is presented with a cue. At this point, 
the user attempts to actively recall the answer. When he has performed this 
task to his satisfaction, the user taps on the "Show the Answer" button shown 
in Fig. 40. 

Fig. 41 shows a further operation of the Review Module 22 including a 

15 Rating Quality of Response. In one preferred embodiment of the present 
invention, after the user has tapped the "Show the Answer* button shown in 
Fig. 40, the user is presented with the correct response to the cue. The user 
compares his response to the correct response displayed and rates the quality 
of his response on a scale of 1 to 5 where 1 is the lowest quality and 5 is the 

20 highest quality. 

Fig. 42 shows an operation of a Test Module 23 according to a 
preferred embodiment of the present invention, including the presentation of 
cue and a rating of the "Feeling of Knowing." In one preferred embodiment of 
the present invention, after the user has chosen a lesson he would like to be 

25 tested on, or the system or administrator has presented the user with a test via 
the Test Module 23, the user is presented with a cue. The user must actively 
recall what he considers to be the correct response. After he has made his 
attempt at such active recall, the user must determine his "feeling of knowing" 
on a scale of 1 to 5, where 1 is "Don't Know", 3 is "Not Sure" 

30 and 5 is "Know It", and 2 and 4 are gradations between the other scores. 
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Fig. 43 shows a further operation of the Test IVIodule 23 including the 
display of a correct response. In one preferred embodiment of the present 
invention, after the user has chosen a feeling of knowing score as described 
above, the user is presented with five alternative forced-choices. The user 
5 must find his answer among the choices and select the correct answer by 
tapping on the screen. 

Fig. 44 shows another operation of the Test Module 23 including the 
display of a correct response. In one preferred embodiment of the present 
invention, after the user has selected what he considers to be the correct 

10 response from among the distracters as described in Fig. 44, the incorrect 
answers are erased leaving only the correct answer. If this is the answer the 
user selected .in the step described in Fig. 43, the answer remains for a 
relatively short period of time, if it is not the answer that the user selected, the 
answer remains for a relatively longer period of time. 

15 Fig. 45 shows an operation of the Test Module 23 including a display of 

test scores. In one preferred embodiment of the present invention, after the 
user has completed a test, he is provided with test scores which include the 
number of items missed, test score, performance score, percent 
underconfident, and percent overconfident. If the user selected an incorrect 

20 response to the cue, the user will be provided with the opportunity to "re-learn" 
that item. If the user chooses "Yes", the items will be presented in a similar 
way as they were the very first time that the user learned these items. 

Fig. 46 shows an operation of the Schedule Module 25 including a 
Schedule Main Window display. In one preferred embodiment of the present 

25 invention, the user can request that the system 10 calculate and maintain a 

schedule for the user via the Schedule Module 25. The user inputs the starting 
date (defaulted to the current day's date) and the ending date, and identifies 
the lessons to be learned and the name of the schedule. Other relevant 
information can be input by the user, the system 10 or other sources. The 

30 system 1 0 then calculates the most effective and efficient schedule of 

Learning, Reviewing, and Testing so that all items are at the highest state of 
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strength and activation possible on the end date. Also shown in Fig. 46 is a 
progress bar which shows where the user is in the schedule compared to 
where he should be (the vertical hash mark) If the user were following the 
schedule initially prescribed by the system. 
5 Fig. 47 shows an operation of the Connect Module 300 including a 

Connect Main Window display. In one preferred embodiment of the present 
invention, the user can connect the system 10 to another similar system, a 
learning device, a computer including a laptop, palmtop and desktop PC, a 
telephone, a personal digital assistant or to another system via a network 

10 connection such as the Internet. In Fig, 47, the Directory on the right is the 

user's directory of lessons. The directory on the left in Fig. 47 represents the : 
directory of the machine that the user is connected to. To transfer lessons 
between the two, the user simply clicks on the lessons in one window and 
drags them into the other window and drops them. The progress bar and 

1 5 status window on the upper left report the progress of the transfer and 
connection. 

Fig. 48 shows an operation of the Create Module 200 including a Create 
Control Panel display. In one preferred embodiment of the present invention, 
the user can create lessons of his own. In Fig. 48, the Create Control Panel is 

20 shown. This is the panel where the user enters the title of the lesson, the 

author, the date of creation, and a summary of the lesson (which also appears 
in the Preview window described in Fig. 28). The user also sets options which 
determine whether the lesson will be shown in color, with sound, and whether 
the questions and answers will be reversed in the Quick Review portion of the 

25 Learn Module 21 . The user closes (and opens) the panel by tapping on the 
tab on the bottom right hand corner of the panel. The user can also open up a 
list of lessons in the Directory by tapping on the down arrow on the Title input 
window. If a lesson is selected in this manner, the user can review the settings 
or modify then save the lesson. 

30 Fig. 49 shows a further operation of the Create Module 200 including a 

Create Main Window display, in a preferred embodiment of the present 
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invention, the user can create lessons on his own. The display shown in Fig. 

49 is provided when the Control Panel in Fig. 49 is closed. 

The user enters the question and answer as shown in this figure by first 

tapping on one of the buttons on the right labeled from 1 to 12, and then 
5 entering the text in the appropriate window. Two additional input windows are 

available — one above the question and one above the answer. These 

windows allow the user to add pronunciation hints or any other information that 

the user would like to include with each item. The buttons on the right appear 

in different colors depending on the state of the question and answer fields. If 
10 the fields are blank, the button is blue. If the fields have data entered, the 

button is green. The button that Is colored red is the question and answer field 

currently displayed. 

The user can change the ordering of items by tapping on the button 

representing item he wishes to move, then tapping on the move button, then 
15 tapping on the position whqre he would like the item moved to. If there was an 

item already filled in the target location, it is moved to where the first item used 

to be. 

Fig. 50 shows the operation of a progress Module 26 including a 
Progress Main Window display. In a preferred embodiment of the present 

20 invention, the user is provided with feedback about his use of the system 10 
via the Progress Module 26. Fig. 50 shows the various numeric and graphical 
feedback provided. In addition, the user can tap on any field displayed. The 
"teacher" character displayed in the bottom right corner of the display will look 
at the field tapped on. will smile or frown based upon the quality of the score, 

25 and will provide advice on how to improve the score in the thought or dialog 
"bubble" above his head. 

Fig. 51 shows the operation of the Help Module 27 including a Help 
Main Window display. In a preferred embodiment of the present invention, the 
user will be provided with textual and graphical help to assist with the use and 

30 operation of the system's features. The user simply taps on the Help button in 
the lower right comer of the Main Window Control Panel. The Help Index 
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appears on the right and the user taps on the area of interest to reveal mpre 
information. The user taps the Close button when the user is through. 

In another preferred embodiment of the present invention, the system 
10 is embodied in a paper based application in the form of a word-a-day 
5 calendar shown in Fig. 52. In this preferred embodiment, the user is presented 
with one new word each day to learn with one set of information. In this case, 
spelling, part of speech, pronunciation, a full definition, and the use of the word 
within a sentence is included. The user is also presented two words for review 
that were very recently learned. The words are presented with a different set 
10 of information than a word presented the first time. In this case: spelling, part 
of speech, pronunciation and a brief definition. 

The user is also presented several words that were learned further in the past 
The words are presented with a different set of infonmation than a word 
presented the first time or words that were very recently learned. In this case: 

15 spelling and brief definition. 

In this preferred embodiment of the present invention, responses 
(definitions of vocabulary words) are to be actively recalled based upon the 
presentation of cues (vocabulary words). This active recall can be 
accomplished by shielding the responses with paper or plastic until active 

20 recall is attempted, by making invisible responses visible with special pens and 
printed inks after recall is attempted, or any number of ways known to those 
skilled in the art. 

Fig. 53 shows a table including a review expansion series for the paper- 
based system. As illustrated in Fig. 53, items learned should be scheduled for 

25 Review based upon on an expanding rehearsal series In order to maintain long 
term retention. Generally speaking, an adaptive system is desired in order to 
maximize the effectiveness and efficiency of the user's time. The. schedule of 
review for each word learned is defined by Fig. 53. Words learned on day 0 
are reviewed on the first following day, the third day after day 0, one week after 

30 day 0, two weeks after day 0, 

one month after day 0 and so on. 




wo 01/50439 PCTAJSOO/35381 

84 

Numerous additional modifications and variations of the present 
invention are possible in light of the above teachings. As noted above, the 
information to be learned, reviewed and tested and the platforms for learning, 
reviewing and testing items is not limited in any sense and can be modified as 
5 desired. Also, various modules of the various preferred embodiments 

described above can be combined in different combinations to define systems 
as desired. Further, the various modules can operate independently of each 
other or can be interactive and adaptive to each other. Many other 
modifications, combinations and changes may be made to the present 
1 0 invention without departing from the scope of the present invention. It is 

therefore to be understood that within the scope of the appended claims, the 
present invention may be practiced other than as specifically described herein. 
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CLAIMS 

1 . A method of facilitating learning comprising the steps of: 

(a) presenting information to be learned to a user so that the 
information to be learned becomes learned information; 

5 (b) presenting the learned information to the user for providing a review 

of the learned information; 

(c) presenting information to the user for testing whether the learned 
information is known by the user; and 

(d) monitoring the user's performance in each of steps (a), (b) and (c); 

10 and 

(e) scheduling future operation of at least one of steps (a), (b) and (c) 
based on the user's performance monitored in step (d). 

2. A method of learning comprising the steps of: 

15 (a) presenting information to be learned to a user so that the 

information to be learned becomes learned information; 

(b) presenting the learned information to the user for providing a review 
of the learned information; 

(c) presenting information to the user for testing whether the learned 
20 information is known by the user; and 

(d) monitoring the user's performance in each of steps (a), (b) and (c); 

and 

(e) modifying the future performance of at least one of steps (a), (b) 
and (c) based on the results of step (d). 

25 

3. The method according to one of claims 1 or 2, further comprising the 
step of prompting the user to provide user preferences relating to the 
information to be learned and the scheduling of future operation of steps (a), 
(b) and (c). 
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4. The method according to claim 3, wherein the user preferences 
include a desired level of learning and retention, user learning time availability, 
type of information the user desires to learn, and changes in the scheduling of 
steps (a), (b) and (c). 

5 

5. The method according to one of claims 1 or 2, wherein the step of 
presenting information to be learned to the user is done such that the user 
learns the information to be learned to a level of automaticity. 

10 6. The method according to one of claims 1 or 2, wherein the 

information to be learned by the user includes at least one of factual 
knowledge, declarative knowledge, factual information, skills, procedural 
knowledge, information objects, knowledge objects, and items of information 
that are desired to be learned initially and retrieved at a later date, information 

15 used to change behavior, information used to change thought processes, and 
information used to increase the ability to learn, review and test other items. 

7. The method according to one of claims 1 or 2, wherein the 
information to be learned by the user is obtained from one of a text source, an 
20 image source, an audible sound source, a computer, the Internet, a 

mechanical device, an electrical device, an optical device, and the real world. 

8. The method according to one of claims 1 or 2, wherein the 
information to be learned by the user is provided by at least one of a learning 

25 system, a user, an administrator, and an external information source. 

9. The method according to one of claims 1 or 2, wherein the 
information to be learned by the user is presented to the user in the form of 
paired associate items each including a cue and a response. 

30 




wo 01/50439 PCT/USOO/35381 

87 

1 0. The method according to one of claims 1 or 2, wherein the 
information to be learned by the user is presented to the user at least one of 
visually, auditorily and kinesthetically. 

5 .11- The method according to one of claims 1 or 2, wherein the 

information to be learned by the user is presented to the user in one of a serial 
order and a non-serial procedural order for skill-based learning. 

12. The method according to one of claims 1 or 2, wherein the step of 
1 0 presenting the learned information to the user for providing a review is 

performed such that the learned information is reviewed and maintained by the 
user at a desired level of long term retention. 

13. The method according to one of claims 1 or 2, further comprising 
15 the step of determining when the learned information has fallen below a 

desired level of retention of the learned information for the user and in 
response thereto performing the step of presentirig the learned information to 
the user for providing a review. 

20 14. The method according to claim 13, further comprising measuring 

the memory strength of the learned information to determine when the learned 
information has fallen below a desired level of retention of the learned 
information. 

25 15. The method according to one of claims 1 or 2, wherein the 

information to be learned by the user is presented to the user in the form of 
paired associate items each including a cue and a response and the 
presentation of cues and responses to the user is performed using a CS-US 
interval and is based on at least one of the type of information, difficulty of the 

30 information to learn, the user's performance in each of the steps (a), (b) and 
(c). measured arousal and attention of the user, the measured confidence of 
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the user in responding to the presentation of cues and providing responses to 
cues, the number of times each of the paired-associate items has been 
presented to the user, the user's feeling of knowing and judgment of knowing 
as quantitatively rated by the user, the measured latency of response of the 
5 user, the measured memory strength for each of the paired associate items, 
and the measured probability of recall for each of the paired associate items. 



16. The method according to one of claim 1 or 2, wherein the step of 
scheduling future operation of steps (a), (b) and (c) is performed to optimize 

10 the learning, memory and performance of the learned information by the user. 

17. The method according to claim 1 , wherein the step of scheduling 
includes calculating a schedule of the presentation of the information in at 
least one of the steps (a), (b) and (c) to maximize the strength and activation of 

15 the learned information by a predetermined date. 

18. The method according to one of claims 1 or 2, wherein the steps of 
the method are embodied as one of computer software, a signal carrier wave 
format to be used on an Internet-based system, machine-executable 

20 instructions, computer-executable instructions for operation on a processor- 
based system including at least one of a computer, a telephone, a personal 
digital assistant and an information transmission device. 

19. The method according to one of claims 1 or 2, wherein the steps 
25 of the method are performed via one of a paper-based system, a computer- 
based system, a human-based system, and a system that presents information 
to a person or organism for learning and future retrieval of the information. 



30 



20. The method according to one of claims 1 or 2, wherein the steps of 
the method are performed via a processor-based system. 
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21. The method according to one of claims 1 or 2, wherein the steps of 
the method are performed via a non-processor-based system. 

22. The method according to one of claims 1 or 2, wherein the steps of 
5 the method are performed via at least one of a processor, a central processing 

unit, a computer, a telephone, a cellular telephone', a personal digital assistant, 
a hand-hetd electronic, device, an information transmission device and a digital 
data/information transmission device which performs the steps via processing 
of instructions embodied in machine readable code or computer executable 
10 code. 



23. The method according to onei of claims 1 or 2, further comprising 
the step of presenting the information to be learned in the form of paired 
associate items each including a cue and a response, and determining at least 

15 one of a presentation pattern, a presentation rate, and a presentation 

sequence of the paired associate items based on the performance of the user 
in at least one of steps (a), (b) and (c). 

24. The method according to one of claims 1 or 2, wherein the steps of 
20 the method are performed via at least one of audio tapes, video tapes, and 

paper-based systems including calendars, books, and documents. 

25. The method according to one of claims 1 or 2, further comprising 
the step of measuring memory strength of the learned information during the 

25 user's operation in at least one of steps (b) and (c). 

26. The method according to claim 25, wherein the step of measuring 
memory strength is performed by gathering overt and covert Information during 
the user's operation in at least one of steps (a), (b) and (c). 
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27. The method according to one of claims 1 or 2, further comprising 
the step of allowing the user to input the information to be learned. 

28. The method according to claim 27, further comprising the step of 
5 allowing the user to input information relating to the information to be learned 

input by the user in the step of allowing the user to input the information to be 



29. The method according to one of claims 1 or 2, further comprising 
10 the step of displaying to the user information reiating to the user's progress 
with at least one of steps (a), (b) and (c). 

30. The method according to one of claims 1 or 2, wherein the step of 
monitoring the user's performance during each of steps (a), (b) and (c) is done 

1 5 without the user knowing about being monitored. 

31 . The method according to one of claims 1 or 2, further comprising 
the step of identifying at least two confusable items to be learned and 
presenting to the user the at least two confusable items to be learned 

20 successively during at least one of steps (a), (b) and (c). 

32. The method according to one of claims 1 or 2, wherein during 
testing, prompting the user to provide a feeling of knowing score indicating 
how confident the user is that an answer is correct. 

25 

33. The method according to one of claims 1 or 2, further comprising 
the step of providing information about the user's performance during testing 
including at least one of the number of items missed, a test score, a 
performance score, percent underconfident and percent overconfident. 



learned. 




wo 01/50439 



PCTAJSOO/35381 



91 

34. The method according to claim 33, further comprising the step of 
inquiring if the user desires to relearn items that the user provided an incorrect 



response to during testing and allowing the user to relearn the items that the 
user provided an incorrect response to during testing. 



35. The method according to one of claims 1 or 2, further comprising 
the steps of presenting the information to be learned and the learned 
information in the form of paired associate items each including a cue and a 
response, and determining a sequence of items to be learned and a time 

1 0 period between a presentation of the cue and the response of each of the 
paired-associate items and a time period between presentation of successive 
paired associate items to achieve a desired level of learning. 

36. The method according to one of claims 1 or 2, wherein the 

15 information to be learned is presented by generating a sequence of known and . 
unI<nown items and presenting the known and unknown items via an expanded 
rehearsal series adapted to the unknown item. 

37. The method according to claim 36, further comprising the step of 
20 adapting the expanded rehearsal series to the user based on the user's 

performance. 

38. The method according to claim 36, wherein the sequence of known 
and unknown items is arranged to provide an intra-trial spacing effect. 



5 



25 



39. The method according to claim 37, wherein the sequence of known 
and unknown items is arranged to provide an inter-trial spacing effect. 



30 



40. The method according to one of claims 1 or 2, further comprising 
the step of allowing the user to interrupt the presenting of the information to be 
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25 



learned and the learned information during at least one of steps (a), (b) and 



41. The method according to one of claims 1 or 2, wherein the 
performance of the user that is measured includes at least one of time required 
for user to input various responses, the ability of user to adhere to a schedule 
of operation of steps (a), (b) and (c), the user's level of interest and arousal in 
learning. 

42. The method according to claim 41 , wherein the user's level of 
interest and arousal in learning is determined by measuring at least one 
physiological characteristic of the user. 

43. The system according to one of claims 1 or 2, further comprising 
the step of calculating for each of the paired associates a memory strength 
over time based on a rate of initial learning, a degree of initial learning, a 
probability of recall, a latency of recall and savings in relearning. 

44. The method according to claim 43, further comprising the steps of 
presenting the information to be learned and the learned information in the 
form of paired associate items each including a cue and a response, wherein 
the calculated memory strength for each of the paired associate items is used 
to adaptively modify the operation of at least one of steps (a), (b) and (c). 

45. The method according to one of claims 1 or 2, further comprising 
the steps of presenting the information to be learned and the learned 
information in the form of paired associate items each including a cue and a 
response and determining when a memory strength for a particular item has 
decreased to a minimum retention level for the user by making calculated 
projections related to at least one of the decline of human memoiy. the type 
and difficulty of each of the paired associate items, the recency of presentation 



(c). 
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of the paired associate item, the frequency of presentation of the paired 
associate item, the pattern of prior user's exposure to the paired associate 
item, and the user's particular history of past performance in operation of steps 
(a), (b) and (c). 

5 

46. The method according to one of claims 1 or 2, further comprising 
the steps of presenting the information to be learned and the learned 
information in the form of paired associate items each including a cue and a 
response, monitoring the memory strength for each of the paired associate 

1 0 items and determining a schedule for review or testing of the paired associate 
items based on the monitored memory strength, 

47. The method according to one of claims 1 or 2, further comprising 
the step of presenting the information to be learned and the learned 

15 information based on an expanded rehearsal series that is adapted to the user 
based on the measured user performance. 

48. The method according to one of claims 1 or 2, further comprising 
the steps of presenting the information to be learned and the learned 

20 information in the form of paired associate items each including a cue and a 
response, wherein the user's performance is measured based on the user's 
ability to evaluate accurately whether the user l<nows the correct response and 
the incorrect responses to the cue of the paired associate items. 

25 49. The method according to claim 48, wherein the step of measuring 

the user's performance includes the step of determining the time required for 
the user to make the evaluation about the correct response and incorrect 
responses. 
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50. The method according to one of claims 1 or 2, further comprising 
the step of determining metacognitive skills of the user by measuring at least 
one of the user's feeling of knowing, confidence, and judgment of learning. 

51 . The method according to one of claims 1 or 2, further comprising 
the step of measuring at least one of the user's feeling of knowing, confidence, 
and judgment of learning and modifying future operation of at least one of the 
steps (a), (b) and (c) based on the results of the step of measuring at least one 
of the user's feeling of knowing, confidence, and judgment of learning. 

52. The method according to one of claims 1 or 2, further comprising 
the step of measuring at least one of the user*s probability of recall, latency of 
response, and savings in reiearning. 

15 53. The method according to claim 52, further comprising the step of 

modifying future operation of at least one of steps (a), (b) and (c) based on the 
step of measuring at least one of the user's probability of recall, latency of 
response, and savings in reiearning. 

20 54. The method according to one of claims 1 or 2, further comprising 

the step of asking the user to indicate a desired degree of initial learning 
including one of familiarity, recognition, recall and automaticity. 

55. The method according to claim 54, further comprising the step of 
25 generating a sequence of items to be learned based on a desired degree of 
initial learning indicated by the user. 

56. The method according to one of claims 1 or 2, further comprising 
the steps of presenting the information to be learned and the learned 
30 information in the form of paired associate items each including a cue and a 
response, wherein a timing, order of presentation and sequence of 
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presentation of each cue and response of the paired associate items is 
interactively determined based on information obtained from the user. 

57. The method according to claim 56, wherein the information 
5 obtained from the user is obtained via input based on covertly obtained user 
performance information and overtly obtained information input by the user. 

58. The method according to claim 56, wherein a timing, order of 
presentation and sequence of each cue and response is interactively 
10 determined based on input information including at least one of information 
concerning rate of presentation of the paired associate items, format of 
presentation of the paired associate items, and sequence of presentation of 
the paired associate items. 

15 59. The method according to one of claims 1 or 2, wherein the 

information to be learned and the learned information is presented to the user 
based on at least one of an initial desired degree of memory strength and a 
desired degree of retention. 



20 60. The method according to one of claims 1 or 2, further comprising 

the step of measuring at least one of probability of recall, latency of recall, and 
savings in relearning during operation of steps (a), (b) and (c), and modifying 
future operation of at least one of steps (a), (b) and (c) based on the results of 
the step of measuring at least one of probability of recall, latency of recall, and 

25 savings in relearning during operation of steps (a), (b) and (c). 

61. The method according to one of claims 1 or 2, further comprising 
the steps of presenting the information to be learned and the learned 
information in the form of paired associate items each including a cue and a 
30 response, and modifying at least one of a pattern of presentation of the paired 
associate items, a sequence of paired associate items and a timing of 
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presentation of the paired associate items based on the results of the step of 
measuring the user's performance during at least one of steps (a), (b) and (c). 

62. The method according to one of claims 1 or 2, further comprising 
5 the step of increasing the user's attention to the information to be learned or 

the learned information by presenting obligatory attention cues to the user. 

63. The method according to one of claims 1 or 2, further comprising 
the steps of presenting the information to be learned and the learned 

10 information in the form of paired associate items each including a cue and a 
response, and increasing the user's attention to the information to be learned 
or the learned information by varying at least one of the presentation pattern of 
the paired associate items, the sequence of presentation of the paired 
associate items, and the timing of presentation of the paired associate items. 

15 

64. The method according to one of claims 1 or 2, further comprising 
the steps of presenting the information to be learned and the learned 
information via paired-associate items each including a cue and a response 
and modifying an order of presentation of the paired associate items so as to 

20 eliminate a serial position effect. 

65. The method according to one of claims 1 or 2, further comprising 
the steps of presenting the information to be learned and the learned 
information in the form of paired associate items each including a cue and a 

25 response and reordering the presentation of the cue and response of the 

paired associate items during future operation of at least one of steps (a), (b) 
and (c). 

66. The method according to one of claims 1 or 2, further comprising 
30 the steps of presenting the information to be learned and the learned 

information in the form of paired associate items each including a cue and a 
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response and generating a sequence of the paired associate items for 
presentation to the user in which the paired associate items that are more 
difficult to learn or remember are presented to the user more times than the 
paired associate items that are less difficult to learn or remember. 

5 

67. The method according to one of claims 1 or 2, further comprising 
the step of prompting the user to make a judgment of learning after the 
information to be learned is presented to the user. 

10 68. The method according to one of claims 1 or 2, further comprising 

the step of providing a preview of the information to be learned before the user 
begins learning in step (a). 

69. The method according to one of claims 1 or 2, further comprising 
15 the steps of presenting the information to be learned and the learned 

information in the form of paired associate items each including a cue and a 
response and providing the user an opportunity to review the paired associate 
items that the user has determined to be difficult to learn after learning the 
paired associate items in step (a). 

20 

70. The method according to one of claims 1 or 2, further comprising 
the steps of presenting the information to be learned and the learned 
information in the form of paired associate items each including a cue and a 
response and determining an ease of initial learning for each of the paired 

25 associate items by measuring how many times each of the paired associate 
items was presented to the user before the user indicated that the user had 
learned the paired associate item. 



71. The method according to claim 70, further comprising the step of 
determining a relative degree of difficulty of learning for each of the paired 
associate items based on the ease of initial learning and determining an 
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appropriate review schedule for the paired associate items based on the 
relative degree of difficulty of each of the paired associate items. 

72. The method according to one of claims 1 or 2, further comprising 
the step of using at least one of hopping tables, prediction curves and 
mathematical correlations to accurately control future performance of the steps 

(a) , (b) and (c). 

73. The method according to one of claims 1 or 2, further comprising 
the step of using a learn hopping table to determine the initial schedule of 
presentation of the learned information and modifying a schedule of 
presentation of the learned information in step (b) based on user performance 
and review curves. 

74. The method according to one of claims 1 or 2, further comprising 
the steps of presenting the information to be learned and the learned 
information in the form of paired associate items each including a cue and a 
response, wherein the learned information is presented to the user during step 

(b) so as to increase memory strength of the paired associate items learned 
later and so as to increase memory activation of items learned earlier. 

75. The method according to one of claims 1 or 2, further comprising 
the steps of presenting the information to be learned and the learned 
information in the form of paired associate items each including a cue and a 
response, wherein the learned information is presented to the user during step 
(b) is arranged based on at least one of degree of difficulty of the paired 
associate items, degree of importance of the paired associate items, memory 
strength of the paired associate items, activation of the paired associate items 
and the user's past performance with at least one of steps (a), (b) and (c). 
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76. The method according to one of claims 1 or 2, further comprising 
the steps of presenting the information to be learned and the learned 
information in the form of paired associate items each including a cue and a 
response and arranging the information to be learned or the learned 

5 information based on at least one of importance of the paired associate items, 
strength of the paired associate items and activation of the paired associate 
items. 

77. The method according to one df claims 1 or 2, further comprising 
1 0 the step of arranging the information to be learned or the learned information 

based on an Nth degree polynomiial smoothing function. 

78. The method according to one of claims 1 or 2, further comprising 
the step of measuring the user's latency of recall while presenting the learned 

15 information in step (b). 

79. The method according to claim 78, further comprising the step of 
stopping the review of learned information based on the measured latency of 
recall. 

20 . 

80. The method according to claim 78, further comprising the step of 
stopping the learned information based on a user's quantitative judgment of 
the adequacy of the user's response. 

25 81 , The method according to one of claims 1 or 2, further comprising 

the steps of presenting the information to be learned and the learned 
Information in the form of paired associate items each including a cue and a 
response and determining the time between presentation of a cue and 
presentation of a response of the paired associate items during step (b) based 

30 on at least one of user input, position of the paired associate item within the 
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parred associate items being reviewed, primary sensory modality, and covert 
data obtained from the user. 

82. The method according, to one of claims 1 or 2, further comprising 
the steps of presenting the Infomiation to be learned and the learned 
information in the fomi of paired associate Items each Including a cue and a 
response and an^anglng the presentation of each cue and response of the 
paired associate Items relative to other cues and responses based on timing, 
sequence, and format of the paired associate Items. 



83. The method according to one of claims 1 or 2, further comprising 
the step of presenting the learned infomiation In step (b) based on at least one 
of a forgetting/retention function and an expanded rehearsal series. 

1 5 84. The method according to one of claims 1 or 2, wherein learned 

information presented during step (b) based on a selected one of a plurality of 
forgetting curves which model the forgetting rate for each item of learned 
information and for the user. 

20 85. The method according to claim 84, wherein the selected one of the 

plurality of forgetting curves is modified during the step (b) to model the 
forgetting rate of the user. 

86. The method according to one of claims 1 or 2, further comprising 
26 the step of detennining a forgetting curve that models the rate of forgetting of 
the learned information over time based on a quality response provided by the 
user. 

87. The method according to one of claims 1 or 2, further comprising 
30 the steps of presenting the information to be learned and the learned 

information in the form of paired associate items each including a cue and a 
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response and modifying model for the rate of forgetting of each of the paired- 
associate items for the user based on the user's performance in at least one of 
steps (a), (b) and (c). 

88. The method according to one of claims 1 or 2, wherein the test 
presented during the step of testing includes one of a test of familiarity, a test 
of recognition, a test of recall, and a test of automaticity. 

89. The method according to one of claims 1 or 2. further comprising 
the step of determining whether the learned infomiation is known to the user 
and to what degree the learned information is known to the user. 

90. The method according to claim 1, further comprising the step of 
measuring latency of response during at least one of steps (a), (b) and (c) and 
modifying future operations of the steps (a), (b). and (c) based on the results of 
the step of measuring latency of response. 

91 . The method according to one of claims 1 or 2, wherein the test 
presented during step (c) is an alternative forced-choice test. 

92. The method according to one of claims 1 and 2, wherein the test 
presented during step (c) is a five alternative forced-choice test in which a user 
must select one of the five alternatives presented in response to a cue. 

93. The method according to one of claims 1 or 2, wherein the user is 
asked if the user knows the answer during the step (c) and is asked to provide 
a feeling of knowing by choosing one of a plurality of ratings relating to how 
sure the user is that the user knows the answer 

94. The method according to claim 93, further comprising the step of 
measuring the time required for the user to provide the feeling of knowing. 
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95. The method according to claim 93, further comprising the step of 
prompting the user to select a correct response from a plurality of responses 
presented to the user and measuring the time required for the user to select 

5 one of the plurality of responses. 

96. The method according to claim 95, further comprising the step of 
asking the user how confident the user is that the response selected by the 
user from the plurality of responses is the conrect response and measuring the 

10 time required for the user to indicate how confident the user is that the 
response selected by the user is correct. 

97. The method according to one of claims 1 or 2, further comprising 
the step of asking the user to provide information about the user's feeling of 

15 knowing and confidence of response during step (c). 

98. The method according to one of claims 1 or 2, further comprising 
the step of arranging the learned information during testing performed in step 
(c) based on at least one of difficulty and confusability of the learned 

20 information. 



99. The method according to one of claims 1 or 2, further comprising 
the step of providing the user with the opportunity to review items for which the 
user provided an incorrect response during testing in step (c). 

25 

1 00. The method according to one of claims A or 2, wherein the learned 
information to be tested in step (c) is arranged and presented to the user 
based on at least one of time of day, gender of the user, age of the user, 
physiological measures of the user, measures of attentiveness of the user. 

.30 environmental conditions, the type of learned information to be tested, difficulty 
of the learned information, the recency of the learned information, the 
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frequency of the learned information and a pattern of prior exposure of the 
user to the learned information. 

101. The method according to one of claims 1 or 2, further comprising 
the step of scheduling the operation of at least one of steps (a), (b) and (c) 
based on at least one of the information input by the user, information input by 
a learning administrator, and infonnatiori input by a learning system 

102. The method according to one of claims 1 or 2, further comprising 
the step of scheduling the operation of at least one of steps (a), (b) and (c) 
based on at least one of date that the learned information must be known to 
the user, the current date, the start date, the information to be learned that 
must be learned by the user between the start date and the test date, the 
user's desired degree of initial learning and retention, times that the user is not 

1 5 available and the user's ability to adhere to previously generated schedules. 

1 03. The method according to claim 1 , wherein during the step of 
scheduling the operation of steps (a), (b) and (c). the information to be learned 
and the learned information is arranged and distributed to achieve workload 

20 smoothing. 

104. The method according to one of claims 1 or 2, further comprising 
the step of scheduling the operation of at least one of steps (a), (b) and (c) 
based on at least one of inputs from the user, inputs from a learning 

25 administrator, the spacing effect, sfrength of the learned information, activation 
of the learned information, when a.lesson was initially learned, the degree of 
difficulty of learned infonriation, and the confusability of learned information. 



30 



105. The method according to one of claims 1 or 2, further comprising 
the step of identifying confusable items from the paired associate items, 
identifying a feature of the confusable items which increases discriminability 
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between the confusable items, and using the feature which increases 
discriminability to practice discriminating between the confusable items. 

1 06. The method according to claim 1 05, further comprising the step of 
presenting the confusable items via a blink comparator which overlays and 
alternatively presents to the user the confusable items in a manner to show 
clear differences between the confusable items. 

107. The method according to claim 105, further comprising the step of 
changing ttie presentation of the confusable paired associate items by 
changing the order of the cue and response in each of the paired associate 
items. 

108. The method according to one of claims 1 or 2, further comprising 
the steps of presenting the information to be learned and the learned 
information via paired-associate items each including a cue and a response 
and determining timing between presentation of the cue and the response for 
each of the paired-associate items. 

109. The method according to one of claims 1 or 2, further comprising 
the step of scheduling the operation of steps (a), (b) and (c) based on at least 
one of a start date, an end date, type of information to be learned, a desired 
level of retention, an amount of time each day that the user Is available, a 
desired number of final reviews, amount of time available for final reviews, the 
user's learning history, and black out days when the user is not available. 

1 1 0. The method according to one of claim 1 and 2, further comprising 
the steps of determining a schedule of operation of steps (a), {b) and (c) 
including determining a nomnal review zone, a final review zone and a 
compression review zone. 
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1 1 1 . The method according to claim 1 . further comph-sing the 'step of 
monitoring the user's ability to comply with the generated schedule and 
modifying the schedule based on the monitored information. 

.. .1 12.. The method according to claim 1 . further comprising the step of 
determining if there is any deviation from the schedule, and recalculating the 
schedule to correct for any deviation determined. 

113. The method according to one of claims 1 or 2, further comprising 
the step of asking the user if the user wants to have the information to be 
learned presented faster or slower in step (a). 

114. Amethodaccordingtooneofclaimsl or2. further comprising the 
step of measuring time required for operation of each of steps (a), (b) and (c). 

1 1 5. A system adapted to perform the method of one of claims 1 to 1 1 3. 



116. The system according to claim 1 1 5, further comprising a progress 
module adapted to determine progress of the user and to display to the user 

20 information relating to the measured progress. 

117. The system according to claim 115. further comprising a connect 
module adapted to connect the system to at least one of another learning 
system, a processor, a computer, a telephone, a personal digital assistant and 

25 another system. 



118. The system according to claim 1 15, further comprising a create 
module adapted to allow a user to create lessons including infonnation to be 
learned. 



30 



wo 01/50439 

PCT/USOO/35381 



106 

119. The system according to claim 115, wherein the steps are 
performed by at least one of a computer, a processor, a central processing 
unit, a personal digital assistant, a hand-held electronic device, a telephone, a 
cellular telephone, a digital data/information transmission device, and a device 

5 which performs the steps via processing of instructions embodied in one of 
machine readable code and computer executable code. 

120. The system according to claim 115, wherein the system is 
adapted to be accessed via at least one of a keyboard, a microphone, a 

1 0 mouse, a touchscreen, a musical keyboard, a telephone, the Internet, and an 
Information transmitting device. 



15 



121. An apparatus adapted to perform the method of one of claims 1 to 
113. 

122. The apparatus according to claim 121, further comprising a 
progress module adapted to determine progress of the user and to display to 
the user information relating to the measured progress. 

20 1 23. The apparatus according to claim 121, further comprising a 

connect module adapted to connect the system to at least one of another 
learning system, a processor, a computer, a telephone, a personal digital 
assistant and another systern. 

25 124. The apparatus according to claim 121 . further comprising a 

create module adapted to allow a user to create lessons including information 
to be learned. 



30 



125. The apparatus according to claim 121, wherein the apparatus is 
one of a computer, a processor, a central processing unit, a personal digital 
assistant, a hand-held electronic device, a telephone, a cellular telephone, a 
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digital data/information transmission device, and a device which performs the 
steps via processing of instructions embodied in.one of machine readable 
code and computer executable code. 

126. The apparatus according to claim 121, wherein the apparatus is 
adapted to be accessed via at least one of a keyboard, a microphone, a 
mouse, a touchscreen, a musical keyboard, a telephone, the Internet, and an 
information transmitting device. 

127. A learning system comprising: 

a learn module adapted to enable a user to encode knowledge or skills 
via a process for creating a memory record; 

a review module adapted to enable a user to store the knowledge or 
skills via a process of maintaining a mernpry record over time through 
rehearsal; and 

a test module is adapted to enable a user to retrieve the knowledge or 
skills via a process of producing a response to a presented cue automatically 
or through active recall; wherein 

the leam module, the review module and the test modules are adapted 
to be interactive with each other and operate such that future operation of each 
of the leam module, the review module and the test module is based on past 
performance in at least one of the other of the learn module, the review module 
and test module. 

128. A learning system comprising: 

a learn module adapted to enable a user to encode knowledge or skills 
via a process for creating a memory record; 

a review module adapted to enable a user to store the knowledge or 
skills via a process of maintaining a memory record over time through 
rehearsal; 
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a test module adapted to enable a user to retrieve the kno\Artedge or 
skills via a process of producing a response to a presented cue automatically 
or through active recall; and 

a schedule module adapted to schedule future operation of each of the 
learn module, the review module and the test module based on past 
performance of the user with at least one of the learn module, the review 
module and the test module. 



129. The system according to one of claims 127 or 128, wherein the 
10 system is one of a paper-based system, a computer-based system, a human- 
based system, a non-processor based system, a processor-based system, and 
a system that presents information to a person or organism for learning and 
■ future retrieval of the information. 

15 1 30. The system according to one of claims 127 or 128, wherein the 

learn module, the review module and the test module are embodied as one of 
computer software, a signal carrier wave format to be used on an Internet- 
based system, machine-executable instructions, computer-executable 
instructions for operation on a processor-based system including at least one 

20 of a computer, a telephone, a personal digital assistant and an information 
transmission device. 



131. The system according to one of claims 127 or 128, wherein the 
system includes at least one of a processor, a central processing unit, a 
25 computer, a telephone, a cellular telephone, a personal digital assistant, a 
hand-held electronic device, an information transmission device and a digital 
data/information transmission device which performs the steps via processing 
of instructions embodied in machine readable code or computer executable 
code. 
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132. The system according to one of claims 127 or 128, wherein the 
learn module, the review module and the test module are embodied as one of 
machine readable code, computer software, computer executable code and a 
signal can-ier wave transmitted via the Internet. 

133. The system according to one of claims 127 or 128, wherein the 
learn module, the review module and the test module are independent of each 
other and are adapted to interact with each other. 

134. The system according to one of claims 127 or 128, further 
comprising at least one of a keyboard, a microphone, a mouse, a touchscreen, 
a musical keyboard, a musical Instrument, a telephone, a Internet connection 
device, and an infomnation transmitting device, adapted to input information 
into the system. 

1 35. The system according to one of claims 127 or 128, further 
comprising an overt information gathering device for overtly obtaining from the 
user information that is used to modify operation of at least one of the learn 
module, the review module and the test module. 

136. The system according to one of claims 127 or 128, further 
comprising a covert Information gathering device for covertly obtaining from 
the user infomiation that is used to modify operation of at least one of the learn 
module, the review module and the test module. 

1 37. The system according to one of claims 127 or 128, further 
comprising an input module for allowing Infomiation that is used to modify 
operation of at least one of the learn module, the review module and the test 
module to be input to the. system. 
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138. The system according to one of claims 127 or 128, further 
comprising a create module adapted to allow a user to create lessons to be 
presented in at least one of the learn module, the review module and the test 
module. 

1 39. The system according to one of claims 1 27 or 1 28. further 
comprising a connect module adapted to allow the system to be connected to 
or communicate with one of a person, another system and an infomiation 
transmission apparatus. 

140. The system according to one of claims 127 or 128, further 
comprising a progress module adapted to measure the user's progress with 
the system and to present information relating to the measured user progress. 

141 . The system according to one of claims 127 or 128, wherein each 
of the learn module, the review module and the test module is.adapted to take 
quantitative measurements relating to the performance of the user with the 
system. 

142. The system according to claim 141, wherein the future operation . 
of at least one of the learn module, the review module and the test module is 
modified based on the quantitative measurements relating to the performance 
of the user with the system. 

143. The system according to claim 141, wherein the quantitative 
measurements measure at least one of latency of recall, recency of 
presentation of information to the user, fi-equency of presentation of 
information to the user, and the pattern of prior exposure of information to the 
user. 
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144. The system according to claim 141. wherein the quantitative 
measurements measure at least one of time for the user to actively recall 
information, memory strength of the user for an item of infomiation, the arousal 
of the user, the attention of the user, the confidence of the user that the user 
5 knows an item of information or has provided a correct response, a 
consolidation period, ease of learning an item of information, feeling of 
knowing, habituation, judgment of learning, probability of recall, savings in 
relearning, sensitization, the spacing effect, time of day effect, and vigilance of 
the user. 



145. The system according to one of claims 127 or 128, wherein each 
of the learn module, the review module and the test module operates 
independently of the others of the learn module, the review module and test 



146. The system according to one of claims 127 or 128, wherein each 
of the learn module, the review module and the test module operates together 
with the others of the learn module, the review module and test module. 

147. The system according to one of claims 127 or 128, wherein the 
learn modufe, the review module and the test module are integrated together 
to define a main engine. 

148. The system according to one of claims 127 or 128, wherein the 
review module is adapted to measure memory strength of a user for an item of 
information, determine when the memory strength will fall below a desired 
memory strength level, and present the item of infonnation for review by the 
user at of after the time that the memory strength will fall below a desired 
memory strength level. 
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149. The system according to one of claims 127 or 128, wherein the 
review module presents items of information to the user using an expanded 
rehearsal series for each of the items of information. 

150. The system according to one of claims 127 or 128, wherein the 
system calculates based on the user performance with at least one of the learn 
module, the review module and the test module, a rate of initial learning, a 
degree of initial learning, the probability of recall, the latency of recall and 
savings in releaming. 

1 51 . The system according to one of claims 127 or 128, further 
comprising an input module adapted to enable a user to input a desired level 
of learning or retention of the knowledge or skills. 

"•^ ""S^. The system according to one of claims 127 or 128, wherein the 

system is adapted to measure a probability of recall, latency.of response, and 
savings in releaming and the system modifies future operation of at least one 
of the learn module, the review module and the test module based on at least 
one of the measured probability of recall, the measured latency of response 

20 and the measured savings in releaming. 

1 53. The system according to one of claims 1 27 or 1 28, v^^erein the 
learn module is adapted to present information to be learned via paired 
associate Items each including a cue and a response. 



25 



154. The system according to one of claim 153, wherein the learn 
module is adapted to vary at least one of the timing, order of presentation and 
sequence of each cue and response of the paired associate items. 



30 



1 55. The system according to claim 1 54, wherein the learn module is 
adapted to vary the at least one of the timing, order of presentation and 
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sequence of each cue and response of the paired associate items based on at 
least one of the user's performance with the learn module, the review module 
or the test module, sensed environmental conditions, infonnation input by the 
user, desired degree of initial learning, desired degree of retention over time 
5 and information relating to the user's past performance with the system. 

156. The system according to claim 154, wherein the system is adapted 
to vary the at least one of the timing, order of presentation and sequence of 
each cue and response of the paired associate items to reduce cognitive 

1 0 workload of the user during learning. 

1 57. The system according to claim 1 54, wherein the system is adapted 
to vary the at least one of the timing, order of presentation and sequence of 
each cue and response of the paired associate items to reduce fatigue of the 

15 user during learning. 

158. The system according to claim 154, wherein the system is adapted 
to vary the at least one of the timing, order of presentation and sequence of 
each cue and response of th^ paired associate items to prevent habituation of 

20 the user. 



159. The system according to one of claims 127 or 128, wherein the 
learn module is adapted to present a plurality of items to be learned to the user 
such that the user learns the items to a level of autbmaticity. 

160. The system according to one of claims 127 or 128. wherein the 
learn module is adapted to present a plurality of items to be learned to the user 
such that the user learns the items to a desired level of learning. 

161 . The system according to one of claims 127 or 128, wherein the 
learn module is adapted to present a plurality of items to be learned using 
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obligatory attention cues including at least one of a blinking light, a tone, and 
object movement. 

162. The system according to one of claims 127 or 128, wherein the 
leam module is adapted to present a plurality of items to be learned in a 
manner to eliminate the serial position affect. 

163. The system according to claim 162, wherein the order of 
presentation of the plurality of items to be learned is changed to eliminate the 
serial position effect, 

164. The system according to one of claims 127 or 128, vi^ierein the 
leam module is adapted to present a group of known Items and unknown items 
to the user and the learn module presents the known Items together with the 
unknown items to the user. 



165. The system according to claim 164, wherein the leam module 
generates the group of known items and unknown items and determines how 
to present the group of known items and unknown items based on at user 
preferences, past performance of the user with the system and how difficult ai 
item is to leam. 

166. The system according to one of claims 127 or 128, wherein the 
leam module is adapted to present items to be teamed to the user as many 
times as is necessary to achieve a desired degree of overlearning. 

167. The system according to claim 166, wherein the learti module is 
adapted to measure the user's latency of recall for each of the items and 
determines when the desired degree of overlearning Is reached based on the 
measured latency of recall. 



10 
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168. The system according to one of claims 127 or 128 wherein the 
learn module is adapted to present a plurality of items to be learned to the user 
and prompt the user to provide a judgment of learning after the user has been 
presented with the plurality of items to be learned. 

169. The system according to one of claims 127 or 128, wherein the 
learn module is adapted to present to the user a preview of the knowledge or 
skills to be learned before the user begins learning. 

170. The system according to one of claims 127 or 128, wherein the 
learn module is adapted to present a plurality of items to be learned to the user 
and provide a quick review to provide the user with an op|i)ortunlty to review 
difficult or unlearned items before the user completes the use of the learn 
module. 



15 



20 



171. The system according to claim 170, wherein the learn module 
reorders the presentation of the plurality of items during the quick review so as 
to eliminate the serial position effects of primacy, recency, and the Von 
Restorff effect. 

172. The system according to claim 170, wherein the learn module 
sorts the plurality of Items during the quick review using a drop out method. 

173. The system according to claim 170, wherein the learn module is 
25 adapted to determine an ease of initial learning of each item based on how 

many times an item was presented to the user before the user indicated that 
the item was known to the user. 

174. The system according to one of claims 127 or 128, wherein the 
30 learn module is adapted to present a plurality of items to be learned to the user 
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and uses a learn hopping table and a plurality of learning curves to determine 
how to present the plurality of items to the user. 

175. The system according to claim 174, wherein the plurality of 

5 learning curves are based on at least one of how difficult an item is to learn, 
how an item is presented to the user, learning rate, degree of retention, type of 
information to be learned, and information about the user. 

176, The system according to claim 174. wherein the learn module is 
10 adapted to select one of the learning curves based on the user's performance 

with the system. 

177. The system according to one of claims 127 or 128, wherein the 
learn module is adapted to measure the user's performance and the system 
1 5 modifies future operation of at least one of the review module and the test 
module based on the measured user performance in the test module. 

178. The system according to one of claims 127 or 128, wherein the 
review module is adapted to present items that have been learned by the user 

20 in the learn module in order to allow the user to review the learned items. 

179. The system according to one of claims 127 or 128, wherein the 
system is adapted to allow the user to select whether to review items that have 
been learned in the learn module. 

25 

180. The system according to one of claims 127 or 128, wherein the 
review module is adapted to present items that have been learned by the user 
in the learn module in a manner based on at least one of a desired level of 
retention of the learned items, memory strength, activation, degree of difficulty 

30 to learn or remember the item, degree of importance of the item, strength, 
activation and how the user has interacted with the system in the past. 
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181. The system according to one of claims 127 or 128, wherein the 
review module is adapted to present items that have been learned by the user 
and in a manner that is arranged based on graceful degradation and workload 
5 smoothing. - 



182. Thesystemaccordingtooneofclaims127or 128, wherein the 
review module is adapted to be accessed by the user at any time via an ad hoc 
review and to be accessed by the user when presented by the system via a 

0 scheduled review. 

183. The system according to one of claims 1 27 or 1 28, wherein the 
review module is adapted to present items that have been learned by the user 
and to measure latency of recall and the user's quantitative judgment of 

5 learning. 

1 84. The system according to claim 183, wherein the review module is 
adapted to stop presenting an item for review based on the measured latency 
of recall and judgment of learning, 

D 

185. The system according to one of claims 127 or 128, wherein the 
review module is adapted to present Items that have been learned by the user 
and in the form of paired associate items each including a cue and a response. 

5 1 86. The system according to claim 1 85, wherein the review module is 

adapted to control the time between presentation of a cue and presentation of 
a response in the review module based on at least one of user input, position 
of the item within sequence of the plurality of items to be reviewed, a primary 
sensory modality, and covertly obtained information from the user. 
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187. The system according to claim 185, wlierein the review module is 
adapted to control the time between each of the, paired associate items based 
on at least one of user input, position of the item within sequence of the 
plurality of items to be reviewed, a primary sensory modality, and covertly 

5 obtained information from the user. 

188. The system according to claim 187, wherein the review module is 
adapted to control the time between each of the paired associate items based 
on at least one of timing of presentation of the paired associate Items, pattern 

1 0 of presentation of paired associate items, sequence of presentation of paired 
associate items, and format of presentation of the paired associate items 
presented in the review module. 



15 



189. The system according to one of claims 127 or 128, wherein the 
review module is adapted to measure the user's perfomiance and the system 
modifies future operation of at least one of the learn module and the test 
module based on the measured user performance in the test module. 

190. The system according to one of claims 127 or 128, wherein the 
20 test module is adapted to provide to the user at least one of a test of familiarity, 

a test of recognition, a test of recall, and a test of automaticity. 

1 91 . The system according to one of claims 127 or 128, wherein the 
test module is adapted to determine to what degree a user knows an item that 

25 has been learned by the user. 



30 



1 92. The system according to one of claims 127 or 128, wherein the 
test module is adapted to present items to be tested and to measure latency of 
response during the presentation of items to be tested. 
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193. The system according to one of claims 127 or 128, wherein the 
test module is adapted to measure the user's performance and the system 
modifies future operation of at least one of the learn module and the review 
module based on the measured user performance in the test module. 

5 

1 94. The system according to one of claims 127 or 128, wherein the 
test module Is adapted to present items to be tested via a multiple forced 
alternative choice test. 

10 1 95. The system according to one of claims 1 27 or 128, wherein the 

test module Is adapted to vary the type of test presented to the user. 

196. The system according to one of claims 127 or 128, wherein the 
test module is adapted to prompt a user to provide a quantitative feeling of 

15 knowing. 

197. The system according to claim 196, wherein the test module is 
adapted to measure the user's latency of response required for the user to 
provide the quantitative feeling of knowing. 

20 

198. The system according to claim 197, wherein the test module is 
adapted to measure relative latency of response based on at least one of the 
differences between measured latency periods, the order of presentation of 
alternative responses, and the primary sensor modality of the items being 
25 tested. 



199. The system according to one of claims 127 or 128. wherein the 
test module is adapted to prompt a user to provide a quantitative confidence of 
response. 
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200. The system according to claim 1 99. wherein the test module is 
adapted to measure the user's latency of response required for the user to 
provide the confidence of response. 

6 201 . The system according to one of claims 1 27 or 128, wherein the 

test module is adapted to present a plurality of items to be tested and in a 
manner such that the most difficult and confusable items are presented before 
less difficult and confusable items are presented. 

10 202. The system according to one of claims 1 27 or 1 28, wherein the 

test module is adapted to present a plurality of items to be tested and to 
measure a user's latency of response during testing, the system modifies 
future operation of at least one of the learn module, the review module and test 
module based on the measured latency of response. 



15 



20 



25 



203. The system according to one of claims 127 or 128, wherein the 
test module is adapted to present a plurality of items to be tested and to 
enable a user to be tested again on items for which the user did not provide 
the correct response. 

204. The system according to claim 203. wherein the test module is 
adapted to provide the user with a review of missed items for which the user 
did not provide the correct response by generating a sequence of the missed 
items based on the relative degree of difficulty of the missed items. 

205. The system according to claim 204. wherein the degree of 
difficulty is determined by the correctness of the user's response and the 
latency of response in providing feeling of knowing and confidence of 
response judgments. 
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206. The system according to one of claims 127 or 128, wherein the 
test module is adapted to present items to be tested as a form of review and 
when the memory strength of each item is relatively high and the activation of 
each item is relatively low. 

207. The system according to one of claims 127 or 128, wherein the 
test module is adapted to determine when a user should be tested based on at 
least one of time of day, gender of the user, age of the user, physiological 
factors of the user, environmental conditions, material to be tested, difficulty of 
material to be tested, recency, frequency and pattern of prior exposure to 
material to be tested. 

208. The system according to one of claims 127 or 128, wherein the 
test module is adapted to present items to be tested as paired associate items 

5 each including a cue and response. 

209. The system according to claim 208, wherein the cue and the 
response of each of the paired associate items are presented in reverse order 
compared to the order of presentation in one of the learn module and the 
review module. 

210. The system according to one of claims 127 or 128, wherein the 
test module is adapted to present items to be tested by presenting a cue and a 
plurality of possible responses including a correct response and a distracter 
that is an incorrect response that is similar to the correct response. 

21 1 . The system according to claim 128, wherein the schedule module 
is adapted to schedule future operation of each of the learn module, the review 
module and the test module based on at least one of quantitative 
measurements of performance of the user, input information from the user, 
input information from an administrator, input infomiation from the system, a 
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test date, a date that knowledge or skills are required, the current date, the 
start date, what knowledge or skills need to be learned between the start date 
and the test date, a desired degree of initial leanning and retention, days that 
study or learning cannot be done, and how closely a person follows the 
5 schedule already created by the system. 

212. The system according to claim 128, wherein the schedule module 
is adapted to schedule presentation of items to be learned, reviewed or tested 
by spreading the presentation of items to reduce cognitive workload and to 

1 0 maximize strength and activation of the items by a predetermined date. 

21 3. The system according to claim 128. wherein the schedule module 
is adapted to schedule all learning, reviewing and testing of knowledge or 
skills to eliminate the need for the user to participate in scheduling. 

15 

21 4. The system according to claim 128, wherein the schedule module 
is adapted to schedule future operation of each of the learn module, the review 
module and the test module based on at least one of the spacing effect, 
strength, activation, when a lesson was initially learned, the degree of difficulty 

20 of items, and the confusability of items: 

215. The system according to claim 128, wherein the schedule module 
is adapted to schedule items for presentation to the user during one of a 
normal zone, a compression zone and a final review zone. 



25 



216. The system according to claim 128, wherein the schedule 
module Is adapted to schedule items for presentation to the user and 
modifies the schedule to achieve workload smoothing and graceful 
degradation. 



30 



217. A learning system comprising: 
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a user interface; 

a learn module for presenting items to be learned via the user interface 
to enable a user to encode knowledge or skills via a process for creating a 
memory record; 

5 means for measuring user performance of the user during learning with 

the learn module; wherein 

the learn module receives data concerning the measured user 
performance from the means for measuring user performance and modifies 
future operation of the learn module based, on the measured past performance 
10 of the user with the learn module. 

218. A learning system comprising: 
a user interface; and 

a learning module adapted to present items to be learned via the user 
1 5 interface to enable a user to encode knowledge or skills via a process for 

creating a memory record, the learning module also being adapted to measure 
user performance during learning while the user is interacting with the learning 
module, and the learning module is adapted to modify future operation of the 
learn module based on the measured past performance of the user with the 
20 learn module. 



219. The system according to one of claims 217 or 218, further 
comprising a schedule module adapted to schedule future operation of learn 
module based on the measured past performance of the user with the learn 

25 module. 

220. The system according to one of claims 21 7 or 21 8. wherein the 
system is one of a paper-based system, a computer-based system, a human- 
based system, a non-processor based system, a processor-based system, and 

30 a system that presents information to a person or organism for learning and 
future retrieval of the information. 
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221 . The system according to one of claims 21 7 or 21 8, wherein the 
learn module is embodied as one of computer software, a signal carrier wave 
format to be used on an Internet-based system, machine-executable 

5 instructions, computer-executable instructions for operation on a processor- 
based system including at least one of a computer, a telephone, a personal 
digital assistant and an information transmission device. 

222. The system according to one of claims 21 7 or 21 8, wherein the 
10 system includes at least one of a processor, a central processing unit, a 

computer, a telephone, a cellular telephone, a personal digital assistant, a 
hand-held electronic device, an information transmission device and a digital 
data/information transmission device which performs the steps via processing 
of instructions embodied In machine readable code or computer executable 
15 code. 

223. The system according to one of claims 217 or 218, wherein the 
learn module is embodied as one of machine readable code, computer 
software, computer executable code and a signal carrier wave transmitted via 

20 the Internet. 



224. The system according to one of claims 217 or 218, further 
comprising at least one of a keyboard, a microphone, a mouse, a touchscreen, 
a musical keyboard, a musical instrument, a telephone, a Internet connection 
25 device, and an Information transmitting device, adapted to Input Information 
into the system. 



225. The system according to one of claims 21 7 or 21 8, further 
comprising an overt information gathering device for overtly obtaining from ttie 
user information that Is used to modify operation of the learn module. 



10 
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226. The system according to one of claims 217 or 218, further 
comprising an covert information gathering device for covertly obtaining from 
the user information that is used to modify operation of the learn module. - 

227. The system according to one of claims 217 or 218, further 
comprising an input module for allowing information that is used to modify 
operation of the learn module to be input to the system. 

228. The system according to one of claims 21 7 or 218, further 
comprising a create module adapted to allow a user to create lessons to be 
presented in the learn module. 



229. The system according to one of claims 217 or 218, further 
comprising a connect module adapted to allow the system to be connected to 
1 5 or communicate with one of a person, another system and an information 
transmission apparatus. 

230. The system according to one of claims 217 or 218, further 
comprising a progress module adapted to measure the user's progress with 
20 the system and to present information relating to the measured user progress. 

231. The system according to one of claims 217 or 218, wherein the 
user performance that is measured includes, quantitative measurements of at 
least one of latency of recall, recency of presentation of infomiation to the 

25 user, frequency of presentation of information to the user, and the pattern of 
prior exposure of information to the user. 

232. The system according to one of claims 217 or 218, wherein the 
user perfonnance that is measured relates to at least one of time for the user 

30 to actively recall information, memory strength of the user for an item of 

information, the arousal of the user, the attention of the user, the confidence of 
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the user that the user knows an item of infonration or has provided a con-ect 
response, a consolidation period, ease of learning an item of information, 
feeling of knowing, habituation, judgment of learning, probability of recall, 
savings in relearning, sensitization, the spacing effect, time of day effect, and 
vigilance of the user. 

233. The system according to one of claims 217 or 218, wherein the 
user performance that is measured includes memory strength of the user for 
each of the items to be learned. 

234. The system according to one of claims 21 7 or 21 8, wherein the 
system calculates based on the user performance with the learn module a rate 
of initial learning, a degree of initial learning, the probability of recall, the 
\atency of recall and savings in relearning. 

235. The system according to one of claims 217 or 218, further 
comprising an input module adapted to enable a user to input a desired level 
of learning or retention of the knowledge or skills. 

236. The system according to one of claims 217 or 218, wherein the 
system is adapted to measure a probability of recall, latency of response, and 
savings in relearning and the system modifies future operation of the learn 
module on at least one of the measured probability of recall, the measured 
latency of response and the measured savings In relearning. 

237. The system according to one of claims 217 or 218. wherein the 
learn module is adapted to present a plurality of paired associate items to be 
learned to the user and to vary at least one of the timing, oreler of presentation 
and sequence of each cue and response of the paired associate items. 
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238. The system according to claim 237. wherein the learn module is 
adapted to vary the at least one of the timing, order of presentation and 
sequence of each cue and response of the paired associate items based on at 
least one of the user's performance with the learn module, sensed 
environmental conditions, information input by the user, desired degree of 
initial learning, desired degree of retention overtime and information relating 
to the user's past performance with the system. 

239. The system according to claim 237, wherein the system is adapted 
to vary the at least one of the timing, order of presentation and sequence of 
each cue and response of the paired associate items to reduce cognitive 
workload of the user during learning. 

240. The system according to claim 237, wherein the system is adapted 
to vary the at least one of the timing, order of presentation and sequence of 
each cue and response of the paired associate items to reduce fatigue of the 
user during learning. 

241. The system according to claim 237, wherein the system is adapted 
to vary the at least one of the timing, order of presentation and sequence of 
each cue and response of the paired associate items to prevent habituation of 
the user. 

242. The system according to one of claims 217 or 218, wherein the 
learn module Is adapted to present a plurality of items to be learned to the user 
such that the user learns the Items to a level of automaticity. 

243. The system according to one of claims 217 or 218, wherein the 
learn module is adapted to present a plurality of Items to be . learned to the user 
such that the user learns the items to a desired level of learning. 
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244. The system according to one of claims 21 7 or 21 8, wherein the 
learn module is adapted to present a plurality of items to be learned using 
obligatory attention cues including at least one of a blinking light, a tone, and 
object movement. 

5 

245. TTie system according to one of claims 217 or 218, wherein the 
learn module is adapted to present a plurality of items to be learned in a 
manner to eliminate the serial position affect. 

1 0 246. The system according to claim 245, wherein the oreJer of 

presentation of the plurality of items to be learned is Ranged to eliminate the 
serial position effect 



247. The system according to one of claims 217 or 218, wherein the 

1 5 learn module is adapted to present a group of known items and unknown items 
to the user and the learn module presents the known items together with the 
unknown items to the user. 

248. The system according to claim 247, wherein the learn module 
20 generates the group of known items and unknown items and determines how 

to present the group of known items and unknown items based on at user 
preferences, past performance of the user with the system and how difficult an 
Item is to learn. 



25 



249. The system according to one of claims 21 7 or 21 8, wherein the 
learn module is adapted to present items to be learned to the user as many 
times as is necessary to achieve a desired degree of overleaming. 



30 



250. The system according to claim 249, wherein the learn module is 
adapted to measure tJie user's latency of recall for each of the items and 
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determines when the desired degree of overleamfng is reached based on the 
measured latency of recall. 

251 . The system according to one of claims 21 7 or 21 8, wherein the 
learn module is adapted to present a plurality of items to be learned to the user 
and prompt the user to provide a judgment of learning after the user has been 
presented with the plurality of items to be learned. 

262. The system according to one of claims 217 or 218, wherein the 
learn module is adapted to present to the user a preview of the knowledge or 
skills to be learned before tha user begins learning. 

253. The system according to one of claims 217 or 21 8. wherein the 
learn module is adapted to present a plurality of items to be learned to the user 
and provide a quick review to provide the user with an opportunity to review 
difficult or unlearned items before the user completes the use of the learn 
module. 



254. The system according to claim 253. wherein the learn module 

20 reorders the presentation of the plurality of items during the quick review so as 
to eliminate the serial position effects of primacy, recency, and the Von 
Restorff effect. 

255. The system according to claim 253, wherein the learn module 
25 sorts the plurality of items during the quick review using a drop out method. 

256. The system according to claim 253. wherein the leani module is 
adapted to determine an ease of initial learning of each item based on how 
many times an item was presented to the user before the user indicated that 

30 the item was kriown to the user. 
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257. The system according to one of claims 217 or 218. wherein the 
learn module is adapted to present a plurality of items to be learned to the user 
and uses a learn hopping table and a plurality of learning curves to determine 
how to present the plurality of items to the user. 

258. The system according to claim 257, wherein the plurality of 
learning curves are based on at least one of how difficult an item is to learn, 
how an item is presented to the user, learning rate, degree of retention, type of 
information to be learned, and information about the user. 

259. The system according to claim 258. wherein the learn module Is 
adapted to select one of the learning curves based on the user's performance 
with the system. 

260. The system according to one of claims.21 7 or 218, wherein the 
learn module is adapted to measure the user's perfonnance and the system 
modifies future operation of other modules based on the measured user 
performance in the learn module. 



20 



261 . A system for allowing a user to store knowledge or skills, the 
system comprising; 

a user Interface; 

a review module adapted to enable a user to store the knowledge 
or skills via a process of maintaining a memory record over time through 
25 rehearsal; 

means for measuring user performance of the user during reviewing 
with the review module; wherein 

the review module receives data concerning the measured user 
performance from the means for measuring user performance and modifies 
30 future operation of the review module based on the measured past 
performance of the user with the review module. 
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262. A system for allowing a user to store knowledge or skills, the 
system comprising: 

a user interface; 

5 a review module adapted to enable a user to store the knowledge 

or skills via a process of maintaining a memory record over time through 
rehearsal, the review module also being adapted to measure user performance 
during reviewing while the user is interacting with the review module, and the 
review module is adapted to modify future operation of the review module 
1 0 based on the measured past perfomnance of the user with the review module. 

263. The system according to one of claims 261 or 262, further 
comprising a schedule module adapted to schedule future operation of review 
module based on the measured past perfbnmance of the user with the review 

15 module. 

264. The system according to one of claims 261 or 262, wherein the 
system is one of a paper-based system, a computer-based system, a human- 
based system, a non-processor based system, a processor-based system, and 

20 a system that presents information to a person or organism for learning and 
future retrieval of the information. 



265. The system accoreJing to one of claims 261 or 262. wherein the 
review module is embodied as one of computer software, a signal carrier wave 
25 format to be used on an Internet-based system, machine-executable 

Instructions, computer-executable instructions for operation on a processor- 
based system including at least one of a computer, a telephone, a personal 
digital assistant and an information transmission device. 

30 266. The system according to one of claims 261 or 262, wherein the 

system includes at least one of a processor, a central processing unit, a 
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computer, a telephone, a cetlular telephone, a personal digital assfstant. a 
hand-held electronic device, an information transmission device and a digital 
data/information transmission device which performs the steps via processing 
of instructions embodied in machine readable code or computer executable 
5 code. 

267. The system according to one of claims 261 or 262, wherein the 
review module is embodied as one of machine readable code, computer 
software, computer executable code and a signal carrier wave transmitted via 
10 the Internet. 

268. The system according to one of claims 261 or 262, further 
comprising at least one of a keyboard, a microphone, a mouse, a touchscreen, 
a musical keyboard, a musical instrument, a telephone, a Internet connection 
device, and an information transmitting device, adapted to input information 
into the system. 



15 



269. The system according to one of claims 261 or 262, further 
comprising an overt information gathering device for overtly obtaining from the 

20 user information that is used to modify operation of the review module. 

270. The system according to one of claims 261 or 262, further 
comprising an covert infomiation gathering device for covertly obtaining from 
the user infomnation that is used to modify operation of the review module. 



25 



271. The system according to one of claims 261 or 262. further 
comprising an input module for allowing information that is used to modify 
operation of the review module to be Input to the system. 
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272. The system according to one of claims 261 or 262, further 
comprising a create module adapted to allow a user to create lessons to be 
presented in the review module. 

273. The system according to one of claims 261 or 262, further 
comprising a conned module adapted to allow the system to be connected to 
or communicate with one of a person, another system and an information 
transmission apparatus. 

274. The system according to one of claims 261 or 262. further 
comprising a progress module adapted to measure the user's progress with 
the system and to present infomiation relating to the measured user progress. 

275. The system according to one of claims 261 or 262, wherein the 
user performance that is measured includes quantitative measurements of at 
least one of latency of recall, recency of presentation of information to the 
user, frequency of presentation of information to the user, and the pattern of 
prior exposure of information to the user. 

276. The system accprding to one of claims 261 or 262, wherein the 
user performance that is measured relates to at least one of time for the user 
to actively recall Information, memory strength of the user for an item of 
information, the arousal of the user, the attention of the user, the confidence of 
the user that the user knows an item of information or has provided a con-ect 
response, a consolidation period, ease of learning an item of information, 
feeling of knowing, habituation, judgment of learning, probability of recall, 
savings in relearning. sensitization, the spacing effect, time of day effect, and 
vigilance of the user. 
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277. The system according to one of claims 261 or 262, wherein the 
user performance that is measured includes memory strength of the user for 
each of the Items being reviewed. 

278. The system according to one of claims 261 or 262. wherein the 
system calculates based on the user performance with the review module a 
rate of Initial learning, a degree of initial learning, the probability of recall, the 
latency of recall and savings in releaming. 

279. The system according to one of claims 261 or 262, further 
comprising an input module adapted to enable a user to input a desired level 
of retention of the knowledge or skills. 

280. The system according to one of claims 261 or 262. wherein the 
system is adapted to measure a probability of recall, latency of response and 
savings in releaming and the system modifies future operation of the review 
module on at least one of the measured probability of recall, the measured 
latency of response and the measured savings in releaming. 

281. The system according to one of claims 261 or 262, wherein the 

review module Is adapted to present a plurality of paired associate items to be 

reviewed to the user and to vary at least one of the timing, order of 

presentation and sequence of each cue and response of the paired associate 
items. 

282. The system according to claim 281, wherein the review module is 
adapted to vary the at least one of the timing, order of presentation and 
sequence of each cue and response of the paired associate items based on at 
least one of the user's performance with the learn module, sensed 
environmental conditions, information input by the user, degree of initial 
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learning, desired degree of retention over time and infomiation relating to the 
user's past performance with the system. 

283. The system according to claim 281 , wherein the system is adapted 
to vary the at least one of the timing, order of presentation and sequence of 
each cue and response of the paired associate items to reduce cognitive 
workload of the user during reviewing. 

264. The system according to claim 281, wherein the system is adapted 
to vary the at least one of the timing, order of presentation and sequence of 
each cue and response of the paired associate items to reduce fatigue of the 
user during reviewing. 

285. The system according to claim 281. wherein the system is adapted 
to vary the at least one of the timing, order of presentation and sequence of 
each cue and response of the paired associate items to prevent habituation of ' 
the user. 

286. The system according to one of claims 261 or 262, wherein the 
review module is adapted to present a plurality of items to be reviewed such 
that the user knows the items to a level of automaticity. 

287. The system according to one of claims 261 or 262, wherein the 
review module is adapted to present a plurality of items to be reviewed to the 
user such that the user reviews the items to achieve a desired level of 
retention. 

288. The system according to one of claims 261 or 262, wherein the 
review module is adapted to present a plurality of items to be reviewed using 
obligatory attention cues including at least one of a blinking light, a tone, and 
object movement. 
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289. The system according to one of claims 261 or 262, wherein the 
review module is adapted to present a plurality of items to be reviewed in a 
manner to eliminate the serial position affect. 

290. The system according to claim 289, wherein the order of 
presentation of the plurality of Items to be reviewed is changed to eliminate the 
serial pbsition effect. 

291 . The system according to one of claims 261 or 262, wherein the 
learn module is adapted to measure the user's latency of recall for each of the 
items and determines when the desired degree of overleaming is reached 
based on the measured latency of recall. 

292. The system according to one of claims 261 or 262, wherein the 
review module is adapted to present a plurality of items to be reviewed to the 
user and uses a review hopping table and a plurality of reviewing cun^es to 
determine how to present the plurality of items to the user. 

293. The system according to claim 292, wherein the plurality of 
reviewing curves are based on at least one of how difficult an item is to learn, 
how an item is presented to the user, learning rate, degree of retention, type of 
information to be learned, and information about the user. 

294. The system according to claim 292, wherein the review module is 
adapted to select one of the review curves based on the user's perfonnance 
with the system. 

295. The system according to one of claims 261 or 262, wherein the 
review module is adapted to measure the user's performance and the system 
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modifies future operation of otiier modules based on the measured user 
performance in thie review module. 

296. The system according to one of claims 261 or 262, wherein the 
review module is adapted to measure memory strength of a user for an item of 
infomiation, determine when the memory strength will fall below a desired 
memory strength level, and present the item of information for review by the 
user at of after the time that the memory strength will fall below a desired 
memory strength level. 

297. The system according to one of claims 261 or 262, wherein the 
review module presents items of Information to the user using an expanded 
rehearsal series for each of the items of information. 

16 298. The system according to one of claims 261 or 262, wherein the 

system calculates based on the user performance with the review module, the 
review module the degree of initial learning, the probability of recall, the 
latency of recall and savings in relearning. 

20 299. The system according to one of claims 261 or 262, further 

comprising an input module adapted to enable a user to input a desired level 
of retention of the knowledge or skills. 

300. The system according to one of claims 261 or 262, wherein the 
25 system is adapted to measure a probability of recall, latency of response, and 
savings in relearning and the system modifies future operation of at least one 
of the review module based on at least one of the measured probability of 
recall, the measured latency of response and the measured savings in 
relearning. 

30 
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301 . The system according to one of claims 261 or 262. wherein the 
system is adapted to allow the user to select whether to review each of the 
items that have been learned. 

6 302. The system according to one of claims 261 or 262, wherein the 

review module is adapted to present items that have been learned by the user 
in a manner based on at least one of a desired level of retention of the learned 
items, memory strength, activation, degree of difficulty to learn or remember 
the item, degree of importance of the item, strength, activation and how the 
1 0 user has interacted with the system in the past. 

303. The system according to one of claims 261 or 262, wherein the 
review module is adapted to present Hems that have been learned by the user 
and in a manner that is arranged to achieve graceful degradation and workload 

15 smoothing. 

304. The system according to one of claims 261 or 262, wherein the 
review module is adapted to be accessed by the user at any time via an ad hoc 
review and to be accessed by the user when presented by the system via a 

20 scheduled review. 

305. The system according to one of claims.261 or 262, wherein the 
review module is adapted to present items that have been learned by the user 
and to measure latency of recall and the user's quantitative judgment of 

25 learning. 

306. The system according to claim 305, wherein the review module is 
adapted to stop presenting an item for review based on the measured latency 
of recall and Judgment of leanning. 



30 
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307. The system according to one of claims 261 or 262, wherein the 
review module is adapted to present items that have been learned by the user 
and in the form of paired associate items each including a cue and a response. 

308. The system according to claim 307, wherein the review module is 
adapted to control the time between presentation of a cue and presentation of 
a response in the review module based on at least one of user input, position 
of the Item within sequence of the plurality of items to be reviewed, a primary 
sensory modality, and covertly obtained information from the user. 



309. The system according to claim 307, wherein the review module is 
adapted to control the time between each of the paired associate items based 
on at least one of user input, position of the item within sequence of the 
plurality of items to be reviewed, a primary sensory modality, and covertly 

15 obtained information from the user. 

310. The system according to claim 307, wherein the review module is 
adapted to control the time between each of the paired associate items based 
on at least one of timing of presentation of the paired associate items, pattern 

20 of presentation of paired associate items, sequence of presentation of paired 
associate items, and format of presentation of the paired associate items 
presented in the review module. 

31 1 . A system for allowing a user to store knowledge or skills, the 
25 system comprising: 

a user interface; 

a test module is adapted to enable a user to retrieve the knowledge or 
skills via a process of producing a response to a presented cue automatically 
or through active recall; wherein a review module adapted to enable a user to 
30 store the knowledge or skills via a process of maintaining a memory record 
over time through rehearsal; 
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means for measuring user performance of thetiser during testing with 
the test module; wherein 

the test module receives data concerning the measured user 
performance from the means for measuring user performance and modifies 
future operation of the test module based on the measured past performance 
of the user with the test module. 

31 2. A system for allowing a user to store knowledge or skills, the 
system comprising: 

a user Interface; 

a test module is adapted to enable a user to retrieve the knowledge or 
skills via a process of producing a response to a presented cue automatically 
or through active recall, the test module also being adapted to measure user 
performance during testing while the user is interacting with the test module, 
and the test module is adapted to modify future operation of the test module 
based on the measured past performance of the user with the test module. 

313. The system according to one of claims 31 1 or 312, further 
comprising a schedule module adapted to schedule future operation of test 
module based on the measured past performance of the user with the test 
module. 



314. The system according to one of claims 31 1 or 312, wherein the 
system is one of a paper-based system, a computer-based system, a human- 
based system, a non-processor based system, a processor-based system, and 
a system that presents information to a person or organism for learning and 
future retrieval of the information. 

315. The system according to one of claims 31 1 or 312. wherein the 
test module is embodied as one of computer software, a signal carrier wave 
format to be used on an Internet-based system, machine-executable 
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instructions, computer-executable instructions for operation on a processor- 
based system including at least one of a computer, a telephone, a personal 
digital assistant and an information transmission device. 

5 316. The system according to one of claims 31 1 or 312, wherein the 

system includes at least one of a processor, a central processing unit, a 
computer, a telephone, a cellular telephone, a personal digital assistant, a 
hand-held electronic device, an infonnation transmission device and a digital 
data/information transmission device which performs the steps via processing 
1 0 of instructions embodied In machine readable code or computer executable 
code. 



317. The system according to one of claims 31 1 or 312, wherein the 
test module is embodied as one of machine readable code, computer software, 
15 computer executable code and a signal carrier wave transmitted via the 
Internet. 

318. The system according to one of claims 31 1 or 312, further 
comprising at least one of a keyboard, a microphone, a mouse, a touchscreen, 

20 a musical keyboard, a musical instrument, a telephone, a Internet connection 
device, and an information transmitting device, adapted to input information 
into the system. 

319. The system according to one of claims 31 1 or 312, further 

25 comprising an overt information gathering device for overtly obtaining from the 
user information that is used to modify operation of the test module, 

320. The system according to one of claims 31 1 or 312, further 
comprising an covert information gathering device for covertly obtaining from 
30 the user information that is used to modify operation of the test module. 
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321 . The system according to one of claims 31 1 or 312, further 
comprising an input module for allowing information that is used to modify 
operation of the test module to be input to the system. 

322. The system according to one of claims 31 1 or 312, further 
comprising a create module adapted to allow a user to create lessons to be 
presented in the test module. 

323. The system according to one of claims 31 1 or 312, further 
comprising a connect module adapted to allow the system to be connected to 
or communicate with one of a person, another system and an infonnation 
transmission apparatus. 

324. The system according to one of claims 31 1 or 312, further 
comprising a progress module adapted to measure the user's progress with 
the system and to present information relating to the measured user progress. 

325. The system according to one of claims 31 1 or 31 2, wherein the 
user performance that is measured includes quantitative measurements of at 
least one of latency of recall, recency of presentation of infonnation to the 
user , frequency of presentation of information to the user, and the pattern of 
prior exposure of information to the user. 

326. The system according to one of claims 31 1 or 312, wherein the 
user perfomiance that is measured relates to at least one of time for the user 
to actively recall infonnation, memory strength of the user for an item of 
infonmation, the arousal of the user, the attention of the user, the confidence of 
the user that the user knows an item of Information or has provided a correct 
response, a consolidation period, ease of learning an item of information, 
feeling of knowing, habituation, judgment of learning, probability of recall, 
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savings in relearning. sensitization, the spacing effect, time of day effect, and 
vigilance of the user. 

327. The system according to one of claims 31 1 or 312, wherein the 
5 user performance that is measured includes memory strength of the user for 

each of the items being test. 

328. The system according to one of claims 31 1 or 312, wherein the 
system calculates based on the user performance with the test module a rate 

10 of initial learning, a degree of initial learning, the probability of recall, the 
latency of recall and savings in relearning. 

329. The system according to one of claims 31 1 or 312, further 
comprising an input module adapted to enable a user to input a desired level 

1 5 of retention of the knowledge or skills. 

330. The system according to one of claims 31 1 or 312, wherein the 
system is adapted to measure a probability of recall, latency of response, and 
savings in relearning and the system modifies future operation of the test 

20 module on at least one of the measured probability of recall, the measured 
latency of response and the measured savings in relearning. 

331. The system according to one of claims 31 1 or 312, wherein the 
test module is adapted to present a plurality of paired associate items to be 

25 tested to the user and to vary at least one of the timing, order of presentation 
and sequence of each cue and response of the paired associate items. 

332. The system according to claim 331 , wherein the test module is 
adapted to vary the at least one of the timing, order of presentation and 

30 sequence of each cue and response of the paired associate items based on at 
least one of the user's performance with the learn module, sensed 
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environmental conditions, infonmation input by the user, degree of initial 
learning, desired degree of retention over time and information relating to the 
user's past performance with the system, 

5 333. The system according to claim 331 , wherein the system is adapted 

to vary the at least one of the timing, order of presentation and sequence of 
each cue and response of the paired associate items to reduce cognitive 
workload of the user during testing. 

10 334. The system according to claim 331, wherein the system is adapted 

to vary the at least one of the timing, order of presentation and sequence of 
each cue and response of the paired associate items to reduce fatigue of the 
user during testing. 

15 335. The system according to claim 331 , wherein the system Is adapted 

to vary the at least one of the timing, order of presentation and sequence of 
each cue and response of the paired associate items to prevent habituation of 
the user, 

20 336. The system according to one of claims 31 1 or 312, wherein the 

test module is adapted to present a plurality of items to be tested such that the 
user knows the items to a level of automaticity. 

337. The system according to one of claims 311 or 312, wherein the 
25 test module is adapted to present a plurality of items to be tested to the user 

such that the user reviews the items to achieve a desired level of retention. 

338. The system according to one of claims 311 or 312, wherein the 
test module is adapted to present a plurality of items to be tested using 

30 obligatory attention cues including at least one of a blinking light, a tone, and 
object movement. 
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339. The system according to one of claims 31 1 or 312, wherein the 
test module is adapted to measure the user's performance and the system 
modifies future operation of other modules based on the measured user 
performance in the test module, 

340. The system according to one of claims 31 1 or 312, wherein the 
test module is adapted to provide to the user at least one of a test of familiarity, 
a tiest of recognition, a test of recall, and a test of automaticity. 

341. The system according to one of claims 311 or 312, wherein the 
test module is adapted to determine to what degree a user knows an item that 
has been learned by the user. 

342. The system according to one of claims 31 1 or 312. wherein the 
test module is adapted to present items to be tested and to measure latency of 
response during the presentation of items to be tested. 

343. The system according to one of claims 31 1 or 312, wherein the 
test module is adapted to measure the user's performance and the system 
modifies future operation of at least one of the learn module and the review 
module based on the measured user performance in the test module. 

344. The system according to one of claims 31 1 or 312, wherein the 
test module is adapted to present items to be tested via a multiple forced 
alternative choice test. 

345. The system according to one of claims 31 1 or 312, wherein the 
test module is adapted to vary the type of test presented to the user. 
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346. The system according to one of claims 31 1 or 312, wherein the 
test module is adapted to prompt a user to provide a quantitative feeling of 
knowing. 

347. The system according to claim 346, wherein the test module is 
adapted to measure the user's latency of response required for the user to 
provide the quantitative feeling of knowing. 

348. The system according to claim 347, wherein the test module is 
adapted to measure relative latency of response based on at least one of the 
differences between measured latency periods, the order of presentation of 
alternative responses, and the primary sensor modality of the items being 
tested. 

349. The system according to one of claims 31 1 or 312, wherein the 
test module is adapted to prompt a user to provide a quantitative confidence of 
response. 

350. The system according to claim 349, wherein the test module is 
adapted to measure the user's latency of response required for the user to 
provide the confidence of response. 

351 . The system according to one of claims 31 1 or 312, wherein the 
test module is adapted to present a plurality of items to be tested and in a 
manner such that the most difficult and confusable items are presented before 
less difficult and confusable items are presented, 

352. The system according to one of claims 31 1 or 312, wherein the 
test module is adapted to present a plurality of items to be tested and to 
measure a user's latency of response during testing, the system modifies 
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future operation of at least one of the learn module, the review module and test 
module based on the measured latency of response. 

353. The system according to one of claims 31 1 or 312, wherein the 
5 test module is adapted to present a plurality of items to be tested and to 

enable a user to be tested again on items for which the user did not provide 
the correct response. 

354. The system according to claim 353, wherein the test module is 
10 adapted to provide the user with a review of missed items for which the user 

did not provide the correct response by generating a sequence of the missed 
items based on the relative degree of difficulty of the missed items. 

355. The system according to claim 354, wherein the degree of 
15 difficulty is determined by the correctness of the user's response and the 

latency of response in providing feeling of knowing and confidence of 
response judgments. 

356. The system according to one of claims 31 1 or 31 2, wherein the 
20 test module is adapted to present items to be tested as a form of review and 

when the memory strength of each item is relatively high and the activation of 
each item is relatively low. 

357- The system according to one of claims 31 1 or 312, wherein the 
25 test module is adapted to determine when a user should be tested based on at 
least one of time of day, gender of the user, age of the user, physiological 
factors of the user, environmental conditions, material to be tested, difficulty of 
material to be tested, recency, frequency and pattern of prior exposure to 
material to be tested. 

30 
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358. The system according to one of claims 31 1 or 312, wherein the • 
test module Is adapted to present items to be tested as paired associate Items 
each including a cue and response. 

5 359. The system according to claim 358, wherein the cue and the 

response of each of the paired associate items are presented in reverse order 
compared to the order of presentation In one of the learn module and the 
review module. 

10 360. The system according to one of clalms S1 1 or 312, wherein the 

test module is adapted to present Items to be tested by presenting a cue and a 
plurality of possible responses including a correct response and a distracter 
that is an Incorrect response that Is similar to the con-ect response. 



15 
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